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Title: Statistical Physics of Hard Optimization Problems 
Author: Lenka Zdeborova 

Abstract: Optimization is fundamental in many areas of science, from computer science 
and information theory to engineering and statistical physics, as well as to biology or social 
sciences. It typically involves a large number of variables and a cost function depending on 
these variables. Optimization problems in the NP-complete class are particularly difficult, 
it is believed that the number of operations required to minimize the cost function is in 
the most difficult cases exponential in the system size. However, even in an NP-complete 
problem the practically arising instances might, in fact, be easy to solve. The principal 
question we address in this thesis is: How to recognize if an NP-complete constraint 
satisfaction problem is typically hard and what are the main reasons for this? We adopt 
approaches from the statistical physics of disordered systems, in particular the cavity 
method developed originally to describe glassy systems. We describe new properties of 
the space of solutions in two of the most studied constraint satisfaction problems - random 
satisfiability and random graph coloring. We suggest a relation between the existence 
of the so-called frozen variables and the algorithmic hardness of a problem. Based on 
these insights, we introduce a new class of problems which we named "locked" constraint 
satisfaction, where the statistical description is easily solvable, but from the algorithmic 
point of view they are even more challenging than the canonical satisfiability. 

Keywords: Constraint satisfaction problems, combinatorial optimization, random color- 
ing problem, average computational complexity, cavity method, spin glasses, replica sym- 
metry breaking, Bethe approximation, clustering of solutions, phase transitions, message 
passing, belief propagation, satisfiability threshold, reconstruction on trees. 
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Titre: Physique statistique des problemes d'optimisation 
Autheur: Lenka Zdeborova 

Resume: L 'optimisation est un concept fondamental dans beaucoup de domaines scien- 
tifiques comme l'informatique, la theorie de 1' information, les sciences de l'ingenieur et 
la physique statistique, ainsi que pour la biologie et les sciences sociales. Un probleme 
d'optimisation met typiquement en jeu un nombre important de variables et une fonc- 
tion de cout qui depend de ces variables. La classe des problemes NP-complets est 
particulierement difficile, et il est communement admis que, dans le pire des cas, un nom- 
bre d'operations exponentiel dans la taille du probleme est necessaire pour minimiser la 
fonction de cout. Cependant, meme ces problemes peuveut etre faciles a resoudre en 
pratique. La principale question considered dans cette these est comment reconnaitre 
si un probleme de satisfaction de contraintes NP-complet est "typiquement" difficile et 
quelles sont les raisons pour cela ? Nous suivons une approche inspiree par la physique 
statistique des systemes desordonnes, en particulier la methode de la cavite developpee 
originalement pour les systemes vitreux. Nous decrivons les proprietes de l'espace des 
solutions dans deux des problemes de satisfaction les plus etudies : la satisfiabilite et 
le coloriage aleatoire. Nous suggerons une relation entre l'existence de variables dites 
"gelees" et la difficulty algorithmique d'un probleme donne. Nous introduisons aussi une 
nouvelle classe de problemes, que nous appelons "problemes verrouilles" , qui presentent 
l'avantage d'etre a la fois facilement resoluble analytiquement, du point de vue du com- 
portement moyen, mais egalement extremement difficiles du point de vue de la recherche 
de solutions dans un cas donne. 

Les mots clefs: Problemes d'optimisation de contraintes, optimisation combinatoire, 
problemes de coloriage, complexite de calcul moyenne, methode de la cavite, verres de 
spins, brisure de la symetrie des repliques, approximation de Bethe, regroupement des so- 
lutions en amas, transitions de phases, passage de messages, propagation des convictions, 
seuil de satisfiabilite, reconstruction sur des arbres. 
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Nazev: Statisticka fyzika tezkych optimalizacnich uloh 
Autor: Lenka Zdeborova 

Abstrakt: Optimalizace je fundament aim koncept v mnoha vednich oborech, pocinaje 
pocitacovou vedou a teorif informace, pfes inzenyrstvi a statistickou fyziku, az po bi- 
ologii ci ekonomii. Optimalizacni uloha se typicky sklada z minimalizace funkce zavisejici 
na velkem mnozstvf promennych. Problemy z takzvane NP-iiplne tridy jsou obzvlaste 
slozite, veri se, ze pocet operacf potrebny k nalezeni fesenf v torn nejtezsmi pripade roste 
exponencialne s poctem promenych. Nicmene i pro NP-iiplne lilohy plati, ze prakticke 
prfpady mohou byt jednoduche. Hlavni otazka, kterou se zabyva tato prace, je: Jak 
rozpoznat, zda je NP-iiplny problem splnitelnosti podminek v typickem pffpade tezky a 
cim je tato slozitost zpusobena? K teto otazce pristupujeme s vyuzitmi znalosti ze stati- 
sticke fyziky neusporadanych, a zejmena skelnych, systemu. Popiseme nove vlastnosti 
prostoru feseni ve dvou z nejvice studovanych optimalizacnich problemu - splnitelnosti 
nahodnych Booleovskych formuli a barvem nahodnych grafu. Navrhneme existenci vz- 
tahu mezi typickou algoritmickou slozitosti a existenci takzvane zamrzlych promennych. 
Na zaklade techto poznatku zkonstruujeme novou tridu problemu, ktere jsme nazvali 
"uzamknute", zde je statisticky popis mnoziny vsech reseni pomerne jednoduchy, ale 
z algoritmickeho pohledu jsou tyto typicke pripady techto problemu jeste tezsi nez v 
kanonickem problemu splitelnosti Booleovskych formuli. 

KHcova slova: Problemy splnitelnosti podminek, kombinatoricka optimalizace, barvem 
nahodnych grafu, prumerna algoritmicka slozitost, metoda kavity, spinova skla, naruseni 
symetrie replik, Betheho aproximace, shlukovani reseni, fazove prechody, posilani zprav, 
propagace domnenek, prah splnitelnosti, rekonstrukce na stromech. 
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Foreword 



P.-G. de Gennes in his foreword to the book "Stealing the gold - A celebration of the 
pioneering physics of Sam Edwards" wrote: 

But he {meaning S. Edwards} also has another passion, which I {meaning P.-G. 
de Gennes} call "The search for unicorns." To chase unicorns is a delicate enterprise. 
Medieval Britons practised it with great enthusiasm (and this still holds up to now: read 
Harry Potter). Sir Samuel Edwards is not far from the gallant knights of the twelfth 
century. Discovering a strange animal, approaching it without fear, then not necessarily 
harnessing the creature, but rapidly drawing a plausible sketch of its main features. 

One beautiful unicorn prancing in the magic garden of Physics has been names "Spin 
glass. " It is rare: not many pure breeds of Spin glasses have been found in Nature. But 
we have all watched the unpredictable jumps of this beast. And we have loved its story - 
initiated by Edwards and Anderson. 

Unicorn is a mythical animal, described in the book of Job, together with another 
strange and fascinating creature which is less peaceful: the leviathan. Leviathans are 
described as immense terrible monsters, invincible beasts. Most people prefer not to 
even think about them. 

This thesis tells a story about what happens when the fierce and mysterious beauty 
of a unicorn meets with the invincibility of a leviathan. 
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Chapter 1 

Hard optimization problems 



In this opening chapter we introduce the constraint satisfaction problems and discuss 
briefly the computer science approach to the computational complexity. We review the 
studies of the random satisfiability problem in the context of average computational com- 
plexity investigations. We describe the connection between spin glasses and random CSPs 
and highlight the most interesting results coming out from this analogy. We explain the 
replica symmetric approach to these problems and show its usefulness on the example of 
counting of matchings [ZDEB-lJ. Then we review the survey propagation approach to 
constraint satisfaction on an example of 1-in-K satisfiability [ZDEB-3]. Finally we sum- 
marize the main contributions of the author to the advances in the statistical physics of 
hard optimization problems, that are elaborated in the rest of the thesis. 

1.1 Importance of optimization problems 

Optimization is a common concept in many areas of human activities. It typically involves 
a large number of variables, e.g. particles, agents, cells or nodes, and a cost function 
depending on these variables, such as energy, measure of risk or expenses. The problem 
consists in finding a state of variables which minimizes the value of the cost function. 

In this thesis we will concentrate on a subset of optimization problems the so-called 
constraint satisfaction problems (CSPs). Constraint satisfaction problems are one of the 
main building blocks of complex systems studied in computer science, information theory 
and statistical physics. Their wide range of applicability arises from their very general 
nature: given a set of N discrete variables subject to M constraints, the CSP consists in 
deciding whether there exists an assignment of variables which satisfies simultaneously 
all the constraints. And if such an assignment exists then we aim at finding it. 

In computer science, CSPs are at the core of computational complexity studies: the 
satisfiability of boolean formulas is the canonical example of an intrinsically hard, NP- 
complete, problem. In information theory, error correcting codes also rely on CSPs. The 
transmitted information is encoded into a codeword satisfying a set of constraints, so that 
the information may be retrieved after transmission through a noisy channel, using the 
knowledge of the constraints satisfied by the codeword. Many other practical problems in 
scheduling a collection of tasks, in electronic design engineering or artificial intelligence 
are viewed as CSPs. In statistical physics the interest in CSPs stems from their close 
relation with the theory of spin glasses. Answering if frustration is avoidable in a system 
is the first, and sometimes highly nontrivial, step in understanding the low temperature 
behaviour. 



1 



2 



CHAPTER 1. HARD OPTIMIZATION PROBLEMS 



A key point is to understand how difficult it is to solve practical instances of a con- 
straint satisfaction problem. Everyday experience confirms that sometimes it is very 
hard to find a solution. Many CSPs require a combination of heuristics and combina- 
torial search methods to be solved in a reasonable time. A key question we address in 
this thesis is thus why and when are some instances of these problems intrinsically hard. 
Answering this question has, next to its theoretical interest, several practical motivations 

• Understanding where the hardness comes from helps to push the performance of 
CSPs solvers to its limit. 

• Understanding which instances are hard helps to avoid them if the nature of the 
given practical problem permits. 

• Finding the very hard problem might be interesting for cryptographic application. 

A pivotal step in this direction is the understanding of the onset of hardness in random 
constraint satisfaction problems. In practice random constraint satisfaction problems 
are either regarded as extremely hard as there is no obvious structure to be explored 
or as extremely simple as they permit probabilistic description. Furthermore, random 
constraint satisfaction models are spin glasses and we shall thus borrow methods from 
the statistical physics of disordered systems. 

1.2 Constraint Satisfaction Problems: Setting 

1.2.1 Definition, factor graph representation 

Constraint Satisfaction Problem (CSP): Consider N variables si...,sjv taking values 
from the domain {0, . . . , q — 1}, and a set of M constraints. A constraint a concerns 
a set of k a different variables which we call da. Constraint a is a function from all 
possible assignments of the variables da to {0, 1}. If the constraint evaluates to 1 we say 
it is satisfied, and if it evaluates to we say it is violated. The constraint satisfaction 
problem consists in deciding whether there exists an assignment of variables which satisfies 
simultaneously all the constraints. We call such an assignment a solution of the CSP. 

In physics, the variables represent g-state Potts spins (or Ising spins if q = 2). The con- 
straints represent very general (non-symmetric) interactions between /c a -tuples of spins. 
In Boolean constraint satisfaction problems (q = 2) a literal is a variable or its negation. 
A clause is then a disjunction (logical OR) of literals. 

A handy representation for a CSP is the so-called factor graph, see |KFL01j for a 
review. Factor graph is a bipartite graph G(V, F, E) where V is the set of variables 
(variables nodes, represented by circles) and F is the set of constraints (function nodes, 
represented by squares). An edge (ia) G E is present if the constraint a G F involves the 
variable % G V. A constraint a is connected to k a variables, their set is denoted da. A 
variable % is connected to Zj constraints, their set is denoted di. For clarity we specify the 
factor graph representation for the graph coloring and exact cover problem in fig. 11.11 
both defined in the following section 11.2.21 

1.2.2 List of CSPs discussed in this thesis 

Here we define constraint satisfaction problems which will be discussed in the following. 
Most of them are discussed in the classical reference book |GJ79j . The most studied 
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Figure 1.1: Example of a factor graph representation for the coloring (left) and the exact 
cover (right) problems. The function nodes (squares) in the graph coloring are satisfied if 
and only if their two neighbours (circles) are in different states (take different colors). The 
function nodes (squares) in the exact cover problem are satisfied if exactly one variable 
(circle) around them takes values 1 (full) and the others (empty). 



constraint satisfaction problems are defined over Boolean variables, q = 2, Si G {0, 1}. 
Sometimes we use equivalently the notation with Ising spins Sj G {— 1,+1}. CSPs with 
Boolean variables that we shall discuss in this thesis are: 

• Satisfiability (SAT) problem: Constraints are clauses, that is logical disjunc- 
tions of literals (i.e., variables or their negations). Example of a satisfiable formula 
with 3 variables and 4 clauses (constraints) and 10 literals: (xi V X2 V -1X3) A (X2 V 
£3) A (-1X1 V -1X3) A (xi V -1X2 V X3). 

• A-SAT: Satisfiability problem where every clause involves K literals, k a = K for 
all a = 1, . . . , M. 

• Not-All-Equal SAT: Constraints are satisfied everytime except when all the lit- 
erals they involve are TRUE or all of them are FALSE. 

• Bicoloring: Constraints are satisfied except when all variables they involve are 
equal. Bicoloring is Not-All-Equal SAT without negations. 

• XOR-SAT: Constraints are logical XORs of literals. 

• Odd (resp. Even) Parity Checks: A constraint is satisfied if the sum of variables 
it involves is odd (resp. even). Odd parity checks are XORs without negations. 

• 1-in-A SAT: Constraints are satisfied if exactly one of the K literals they involve 
is TRUE. 

• Exact Cover, or positive 1-in-A SAT: Constraints are satisfied if exactly one 
of the K variables they involve is 1 (occupied). Exact cover, or positive 1-in-A 
SAT, is 1-in-A SAT without negations. 

• Perfect matching: Nodes of the original graph become constraints, variables 
are on edges and determine if the edge is or is not in the matching, see fig. 11.51 
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Constraints are satisfied if exactly one of the K variables they involve is 1 (belongs 
to the matching). Note that perfect matching is just a variant of the Exact Cover 

• Occupation problems are defined by a binary (K + 1) component vector A. All 
constraints involve K variables, and are satisfied if the sum of variables they involve 
r = J2 da Si is such that A r — 1. 

• Locked Occupation Problems (LOPs): If the vector A is such that AiA i+ i = 
for alH = 0, . . . , K — 1, and all the variables are present in at least two constraints. 

We will also consider in a great detail one CSP with g-ary variables: The graph 
coloring with q colors: Every constraint involves two variables and is satisfied if the two 
variables are not assigned the same value (color). In physics the g-ary variables are called 
Potts spins. 



1.2.3 Random factor graphs: definition and properties 

Given a constraint satisfaction problem with N variables and M constraints, the con- 
straint density is defined as a = M/N. Denote by lZ{k) the probability distribution of 
the degree of constraints (number of neighbours in the factor graph), and by Q{1) the 
probability distribution of the degree of variables. The average connectivity (degree) of 
constraints is 

oo 

K = k = kn i k ) • (1-1) 
The average connectivity of variables is 

oo 

c=i=j2m)- (i-2) 

z=o 

The constraint density is then asymptotically 

M 1 c 

a = — = = = — . 1.3 

N k K K J 

A random factor graph with a given N and M is then created as follows: Draw a 
sequence {h, ■ ■ ■ ,In} of N numbers from the distribution Q(l). Subsequently, draw a 
sequence {k±, . . . , /cm} of M numbers from the distribution 7Z(k), such that J^fli ^ = 
X)i=i h- The random factor graph is drawn uniformly at random from all the factor graphs 
with iV variables, M constraints and degree sequences {/i, . . . , In} and {k±, . . . , /cm}- 

Another definition leading to a Poissonian degree distribution is used often if the 
degree of constraints is fixed to K and the number of variables is fixed to N. There are 
(^) possible positions for a constraint. Each of these positions is taken with probability 



cN 



P=^77m- (1-4) 



The number of constraints is then a Poissonian random variable with average M = cN/ K. 
The degree of variables is distributed according to a Poissonian law with average c 

Q(0 = e"^. (1.5) 
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If K = 2 these are the random Erdos-Renyi graphs [ER59] . This definition works also if 
constraints are changed for variables, that is if the degree of variables and the number of 
constraints are fixed, as in e.g. the matching problem. 

The random factor graphs are called regular if both the degrees of constraints and 
variables are fixed, TZ(k) = 5(k — K) and Q(l) = 5(1 — L). In section POl we will also use 
the truncated Poissonian degree distribution 

l<l: Q(Z) = 0, (1.6a) 

The average connectivity for the truncated Poissonian distribution is then 

e c - 1 

l = C / . —rr . (1.7) 

e c — (c + 1) 

In the cavity approach, the so-called excess degree distribution is a crucial quantity. 
It is defined as follows: Choose an edge (ij) at random and consider the probability 
distribution of the number of neighbours of % except j. The variables (analogously for 
constraints) excess degree distribution thus reads 

(1 + 1)0(1 + 1) m (H1MH1) 
q[l) = = , r(k) = = . (1.8) 

We will always deal with factor graphs where K and c are of order one, and N — ► 
oo, M — » oo. These are called sparse random factor graphs. Concerning the physical 
properties of sparse random factor graphs the two definitions of a random graph with 
Poissonian degree distribution are equivalent. Some properties (e.g. the annealed aver- 
ages) can however depend on the details of the definition. 

The tree-like property of sparse random factor graphs — Consider a random 
variable % in the factor graph. We want to estimate the average length of the shortest 
cycle going through variable %. Consider a diffusion algorithm spreading into all direction 
but the one it came from. The probability that this diffusion will arrive back to % in d 
steps reads 

l \ E- =1 (7(7fc) j 



1 + - F J • («) 

where 7; = I 2 /I — I and 7^ = k 2 /k — l are the mean values of the excess degree distribution 
( 11. 8j) . The probability ( 11.9R is almost surely zero if 

loeiV 

rf< !OgiV_ 
log 7i7 fc 

An important property follows: As long as the degree distributions TZ(k) and Q(l) have a 
finite variance the sparse random factor graphs are locally trees up to a distance scaling 
as logiV ( ll.lOj) . We define this as the tree-like property. 

In this thesis we consider only degree distributions with a finite variance. A general- 
ization to other cases (e.g. the scale-free networks with long-tail degree distributions) is 
not straightforward and many of the results which are asymptotically exact on the tree- 
like structures would be in general only approximative. We observed, see e.g. fig. 12.21 



6 



CHAPTER 1. HARD OPTIMIZATION PROBLEMS 



that many of the nontrivial properties predicted asymptotically on the tree-like graphs 
seems to be reasonably precise even on graphs with about N = 10 2 — 10 4 variables. It 
means that the asymptotic behaviour sets in rather early and does not, in fact, require 
logiV > 1. 

1.3 Computational complexity 
1.3.1 The worst case complexity 

Theoretical computer scientists developed the computational complexity theory in order 
to quantify how hard problems can be in the worst possible case. The most important 
and discussed complexity classes are the P, NP and NP-complete. 

A problem is in the P (polynomial) class if there is an algorithm which is able to 
solve the problem for any input instance of length N in at most cN k steps, where k 
and c are constants independent of the input instance. The formal definitions of what 
is a "problem", its "input instance" and an "algorithm" was formalized in the theory of 
Turing machines [Pap94| , where the definition would be: The complexity class P is the set 
of decision problems that can be solved by a deterministic Turing machine in polynomial 
time. A simple example of polynomial problem is sorting a list of N real numbers. 

A problem is in the NP class if its instance can be stored in memory of polyno- 
mial size and if the correctness of a proposed result can be checked in polynomial time. 
Formally, the complexity class NP is the set of decision problems that can be solved 
by a non-deterministic Turing machine in polynomial time |Pap94| , NP stands for non- 
deterministic polynomial. Whereas the deterministic Turing machine is basically any of 
our today computers, the non-deterministic Turing machine can perform unlimited num- 
ber of parallel computations. Thus, if for finite N there is a finite number of possible 
solutions all of them can be checked simultaneously. This class contains many problems 
that we would like to be able to solve efficiently, including the Boolean satisfiability prob- 
lem, the traveling salesman problem or the graph coloring. Problems which do not belong 
to the NP class are for example counting the number of solutions in Boolean satisfiability, 
or the random energy model [Der80t IDer81j . 

All the polynomial problems are in the NP class. It is not known if all the NP problems 
are polynomial, and it is considered by many to be the most challenging problem in 
theoretical computer science. It is also one of the seven, and one of the six still open, 
Millennium Prize Problems that were stated by the Clay Mathematics Institute in 2000 (a 
correct solution to each of these problems results in a $1,000,000 prize for the author). A 
majority of computer scientists, however, believes that the negative answer is the correct 
one |Gas02] . 

The concept of NP-complete problems was introduced by Cook in 1971 |Coo 71J. All 
the NP problems can be polynomially reduced to any NP-complete problem, thus if 
any NP-complete problem would be polynomial then P=NP. Cook proved [Coo71] that 
the Boolean satisfiability problem is NP-complete. Karp soon after added 21 new NP- 
complete problems to the list |Kar72j . Since then thousands of other problems have 
been shown to be NP-complete by reductions from other problems previously shown to 
be NP-complete; many of these are collected in the Garey and Johnson's "Guide to 
NP-Completeness" [GT79] . 

Schaefer in 1978 proved a dichotomy theorem for Boolean (q = 2) constraint satis- 
faction problems. He showed that if the constraint satisfaction problem has one of the 
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following four properties then it is polynomial, otherwise it is NP-complete. (1) All con- 
straints are such that Sj = 1 for all i is a solution or Sj = for all i is a solution. (2) All 
constraints concern at most two variables (e.g. in 2-SAT). (3) All constraints are linear 
equations modulo two (e.g. in XOR-SAT). (4) All constraints are the so-called Horn 
clauses or all of them are the so-called dual Horn clauses. A Horn clause is a disjunction 
of variables such that at most one variable is not negated. A dual Horn clause is when at 
most one variable is negated. A similar dichotomy theorem exists for 3-state variables, 
q = 3, [Bul02] . Generalization for q > 3 is not known. 

1.3.2 The average case hardness 

Given the present knowledge, it is often said that all the polynomial problems are easy 
and all the NP-complete problems are very hard. But, independently if P=NP or not, 
even polynomial problems might be practically very difficult, and some (or even most) 
instances of the NP-complete problems might be practically very easy. 

An example of a still difficult polynomial problem is the primality testing, a first 
polynomial algorithm was discovered by [AKS04J. But a "proof of remaining difficulty 
is the EFF prize |EFF] of $100,000 to the first individual or group who discovers the first 
prime number with at least 10,000,000 decimal digits. 

And how hard are the NP-complete problems? One way to answer is that under 
restrictions on the structure an NP-complete problem might become polynomial. Maybe 
the most famous example is 4-coloring of maps (planar factor graphs) which is polynomial. 
Moreover, it was a long standing conjecture that every map is colorable with 4 colors, 
proven by Appel and Haken |AH77bl IAH77aj . Interestingly enough 3-coloring of maps is 
NP-complete [GJ79] . 

But there are also settings under which the problem stays NP-complete and yet almost 
every instance can be solved in polynomial time. A historically important example is the 
Boolean satisfiability where each clause is generated by selecting literals with some fixed 
probability. Goldberg introduced this random ensemble and showed that the average 
running time of the Davis-Putnam algorithm [DP601 IDLL62] is polynomial for almost 
all choices of parameter settings |Gol79l [GPB82J. Thus in the eighties some computer 
scientist tended to think that all the NP-complete problems are in fact on average easy 
and it is hard to find the evil instances which makes them NP-complete. 

The breakthrough came at the beginning of the nineties when Cheeseman, Kanefsky 
and Taylor asked "Where the really hard problems are?" in their paper of the same 
name |GKT91j . Shortly after Mitchell, Selman and Levesque came up with a similar 
work [MSL92J . Both groups simply took a different random ensemble of the satisfiability 
(in the second case) and coloring (in the first case) instances: the length of clauses is 
fixed to be K and they are drawn randomly as described in sec. 11.2.31 They observed 
that when the density of clauses a = M/N is small the existence of a solution is very 
likely and if a is large the existence of a solution is very unlikely. And the really hard 
instances were located nearby the critical value originally estimated to be a s m 4.25 in the 
3-SAT [MSL92J. The hardness was judged from the median running time of the Davis- 
Putnam-Logemann-Loveland (DPLL) backtracking-based algorithm [ DP601 IDLL 62J. see 
fig. 11.21 This whipped away the thoughts that NP-complete problems might in fact be 
easy on average. Many other studies and observations followed. The hard instances of 
random .fT-satisfiability became very fast important benchmarks for the best algorithms. 
Moreover, there are some indications that critically constrained instances might appear 
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Figure 1.2: The easy-hard-easy pattern in the random 3-SAT formulas as the constraint 
density is changed. Full lines are probabilities that a formula is satisfiable. Dashed lines 
is the medium running time of the DPLL algorithm. This figure is courtesy of Riccardo 
Zecchina. 



in real-world applications. One may imagine that in a real world situation the amount 
of constraints is given by the nature of the problem, and variables usually correspond to 
something costly, thus the competitive designs contain the smallest possible number of 
variables. 




5.5 4 4.5 5 5.5 4 4.05 4.1 4.15 4.2 4.25 4.3 



density of constraints density of constraints 

Figure 1.3: Probability that a random 3-SAT formula is satisfiable as a function of 
the constraint density. In the inset on the left figure is the position of the crossing 
point between curves corresponding to different sizes as a function of 1/N. It seems to 
extrapolate to the analytical value a s = 4.267 [MZ02L [MMZ06J. This figure should be 
put in contrast with fig. 14.11 where the same plot is presented for the freezing transition 
with a much smaller size of the inset. 

Given a random i^-SAT formula of N variables the probability that it is satisfiable, 
plotted in fig. 11.31 for 3-SAT, becomes more and more like a step-function as the size N 
grows. An analogy with phase transitions in physics cannot be overlooked. The existence 
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and sharpness of the threshold were partially proved [Fri99j. The best known probabilistic 
bounds of the threshold value in 3-SAT are 3.520 for the lower bound |KKL03UHS03] and 
4.506 for the upper bound [DBM00J. Numerical estimates of the asymptotic value of the 
threshold are a s w 4.17 [K594] . a s » 4.258 [CA96j . a s » 4.27 [MZK+99bl lMZK+99aj . 
The finite size scaling of the curves in fig. 11.31 is quite involved as the crossing point is 
moving. That is why the early numerical estimates of the threshold were very inaccurate. 
The work of Wilson [Wil02] , moreover, showed that the experimental sizes are too small 
and the asymptotic regime for the critical exponent is not reached in any of the current 
empirical works. The study of XOR-SAT indeed shows a crossover in the critical exponent 
at sizes which are not accessible for i^-SAT |LRTZ 01j. 

The studies of random if-SAT opened up the exciting possibility to connect the 
hardness with an algorithm-independent property, like the satisfiability phase transition. 
But what exactly makes the instances near to the threshold hard remained an open 
question. 

1.4 Statistical physics comes to the scene 
1.4.1 Glance on spin glasses 

Spin glass is one of the most interesting puzzles in statistical physics. An example of a 
spin glass material is a piece of gold with a small fraction of iron impurities. Physicist, 
on contrary to the rest of the human population, are interested in the behaviour of these 
iron impurities and not in the piece of gold itself. A new type of a phase transition was 
observed from the high temperature paramagnetic phase to the low temperature spin 
glass phase, where the magnetization of each impurity is frozen to a non-zero value, but 
there is no long range ordering. More than 30 years ago Edwards and Anderson |EA75j 
introduced a lattice model for such magnetic disordered alloys 



where Si G { — 1,+1} are Ising spins on a 3-dimensional lattice, the sum runs over all 
the nearest neighbours, h is the external magnetic field and the interaction Jjj is random 
(usually Gaussian or randomly ±J). The solution of the Edwards- Anderson model stays 
a largely open problem even today. 

The mean field version of the Edwards- Anderson model was introduced by Sherrington 
and Kirkpatrick |SK75j . the sum in the Hamiltonian (11.111) then runs over all pairs (ij) 
as if the underlying lattice would be fully connected. Sherrington and Kirkpatrick called 
their paper "Solvable Model of a Spin-Glass". They were indeed right, but the correct 
solution came only five years later by Parisi |Par80c[ IPar80bt IPar80a] . Parisi's replica 
symmetry breaking (RSB) solution of the Sherrington-Kirkpatrick model gave rise to a 
whole new theory of the spin glass phase and of the ideal glass transition in structural 
glasses. The exactness of the Parisi's solution was, however, in doubt till 2000 when 
Talagrand provided its rigorous proof |Tal06j. The relevance of the RSB picture for the 
original Edwards-Anderson model is widely discussed but still unknown. 

A different mean field version of the Edwards-Anderson model was introduced by 
Viana and Bray [VB85], the lattice underlying the Hamiltonian (II. lip is then a random 
graph of fixed average connectivity. The complete solution of the Viana-Bray model is 
also still an open problem. 




(1.11) 
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1.4.2 First encounter 

The Viana-Bray model of spin glasses can also be viewed as random graph bi-partitioning 
(or bi-coloring at a finite temperature). The peculiarity of the spin glass phase will surely 
have some interesting consequences for the optimization problem itself. Indeed, the close 
connection between optimization problems and spin glass systems brought forward a 
whole collection of theoretical tools to analyze the structural properties of the optimiza- 
tion problems. 

All started in 1985 when Mezard and Parisi realized that the replica theory can be 
used to solve the bipartite weighted matching problem [MP85J. Let us quote from the 
introduction of this work: "This being a kind of pioneering paper, we have decided to 
present the method {meaning the replica method} on a rather simple problem (a polyno- 
mial one) the weighted matching. In this problem one is given 2N points i — 1, . . . , 2N, 
with a matrix of distance Uj, and one looks for a matching between the points (a set of 
N links between two points such that at each point one and only one link arrives) of a 
minimal length. " Using the replica symmetric (RS) approach they computed the average 
minimal length, when the elements of the matrix lij are random identically distributed 
independent variables. 

Shortly after Fu and Anderson [FA86] used the replica method to treat the graph 
bi-partitioning problem. They were the first to suggest that, possibly, the existence of a 
phase transition in the average behaviour will affect the actual implementation and per- 
formance of local optimization techniques, and that this may also play an important role 
in the complexity theory. Only later, such a behaviour was indeed discovered empirically 
by computer scientists |CKT91l [MST92] . 

The replica method also served to compute the average minimal cost in the random 
traveling salesmen problem [MP86al MP86bJ. Partitioning a dense random graph into 
more than two groups and the coloring problem of dense random graphs were discussed in 
[KS87J. Later some of the early results were confirmed rigorously, mainly those concerning 
the matching problem [Al dOlj IL W04j . All these early solved models are formulated on 
dense or even fully connected graph. Thus the replica method and where needed the 
replica symmetry breaking could be used in its original form. Another example of a "fully 
connected" optimization problem which was solved with a statistical physics approach is 
the number partitioning problem |Mer98l [Mer OOj . 

And what about our customary random X-satisfiability, which is defined on a sparse 
graph? Monasson and Zecchina worked out the replica symmetric solution in [MZ96, 
MZ97J. It was immediately obvious that this solution is not exact as it largely overes- 
timates the satisfiability threshold, the replica symmetry has to be broken in random 
K-SAT. 

An interesting observation was made in [MZK + 99b] : They defined the backbone of 
a formula as the set of variables which take the same value in all the ground-state con- 
figurations Q. No extensive backbone can exist in the satisfiable phase in the limit of 
large N. If it would, then adding an infinitesimal fraction of constraints would almost 
surely cause a contradiction. At the satisfiability threshold an extensive backbone may 
appear. The authors of [MZK + 99~b] suggested that the problem is computationally hard 
if the backbone appears discontinuously and easy if it appears continuously. They sup- 
ported this by replica symmetric solution of the SAT problem with mixed 2-clauses and 

^dn CSPs with a discrete symmetry, e.g. graph coloring, this symmetry has to be taken into account 
in the definition of the backbone. 
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3-clauses, the so-called 2 +p-SAT. Even if the replica symmetric solution is not correct in 
random i^-SAT and even if it overlooks many other important phenomena the concept 
of backbone is fruitful and we will discuss its generalization in chapter HI 

How to deal with the replica symmetry breaking on a sparse tree-like graph was an 
open question since 1985, when Viana and Bray [VB85J introduced their model. The solu- 
tion came only in 2000 when Mezard and Parisi published their paper "Bethe lattice spin 
glass revisited" [M POlj . They showed how to treat correctly and without approximations 
the first step of replica symmetry breaking (1RSB) and described how, in the same way, 
one can in principal deal with more steps of replica symmetry breaking, this extension is 
however numerically very difficult. But before explaining the 1RSB method we describe 
the general replica symmetric solutions. And illustrate its usefulness on the problem of 
counting matchings in graphs [ZDEB-1]. Only then we describe the main results of the 
1RSB solution and illustrate the method in the l-in-.fr SAT problem [ZDEB-3]. After 
we list several "loose ends" which appeared in this approach. Finally we summarize the 
main contribution of this thesis. This will be the departure point for the following part 
of this thesis which contains most of the original results. 

1.5 The replica symmetric solution 

The replica symmetric (RS) solution on a locally tree-like graph consists of two steps: 

(1) Compute the partition sum and all the other quantities of interest as if the graph 
would be a tree. 

(2) The replica symmetric assumption: Assume that the correlations induced by long 
loops decay fast enough, such that this tree solution is also correct on the only 
locally tree-like graph. 

Equivalent names used in literature for the replica symmetric solution are Bethe-Peierls 
approximation (in particular in the earlier physics references) or belief propagation (in 
computer science or when using the iterative equation as an algorithm to estimate the 
marginal probabilities - magnetizations in physics). Both these conveniently abbreviate 
to BP. 

1.5.1 Statistical physics description 

Let <fi a (da) be the evaluating function for the constraint a depending on the variables 
neighbourhooding with a in the factor graph G(V, F, E). A satisfied constraint has 
<f) a (da) = 1 and violated constraint <fi a (da) = 0. The Hamiltonian can then be writ- 
ten as 

M 

Hg({s}) = E [1 - Mda)} . (1.12) 

o=l 

The energy cost is thus one for every violated constraint. The corresponding Boltzmann 
measure on configurations is: 

MM,/3) = ^e"^« s » , (1.13) 
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where (3 is the inverse temperature and Zc{f3) is the partition function. The marginals 
(magnetizations) xl- are defined as the probabilities that the variable % takes value Sj 



{s ] },j=l,...,i-l,i+l,...,N 



e -/3^G({sj},si) _ (i-i4) 



The goal is to compute the internal energy Eq{(3) and the entropy Sg(/3)- For f3 — > oo 
(zero temperature limit) these two quantities give the ground state properties. We are 
interested in the "thermodynamic" limit of large graphs (N — > oo), and we shall compute 
expectations over ensembles of graphs of the densities of thermodynamical potentials 
e(P) = K[Eg(P)]/N and s((3) = E[Sg((3)]/N, as well as the average free energy density 

fffi = ^[log Z Q {0)\ = ^E[F G (/3)] = e((3) - i 8 (j3) . (1.15) 

The reason for this interest is that, for reasonable graph ensembles, Fg{/3) is self-averaging. 
This means that the distribution of Fc{f3)/N becomes more and more sharply peaked 
around f{(3) when N increases. 

1.5.2 The replica symmetric solution on a single graph 




Oi 



Figure 1.4: Parts of the factor graph used to compute ip^ 1 and x. 



■3- 



•j 



First suppose that the underlying factor graph is a tree, part of this tree is depicted 
in fig. 11.41 We define messages ip^ 1 as the probability that node i takes value S{ on a 
modified graph where all constraints around i apart a were deleted, and xi~* a as t ne prob- 
ability that variable j takes value Sj on a modified graph obtained by deleting constraint 
a. On a tree these messages can be computed recursively as 



VST* = E MM,*,® II = Mix j ^ a }) , (1.16a) 

{sj},j£da—i j&da—i 

l — II ti? = FxM^}) , (l-16b) 



bdaj—a 



where Z a ~^* and Z- 7 "* are normalization constants, the factor (f) a ({s},/3) = 1 if the con- 
straint a is satisfied by the configuration {s} and a ({s},/3) = e _/3 if not. We denote by 
ip a ^ 1 the whole vector (4>q^\ . . . , V'Jpi ) and analogically x^ a — (Xo^ a > • • • j Xg-f)- This 
is one form of the belief propagation (BP) equations [KFLOlt IPea82j . sometimes called 
sum-product equations. The probabilities ip, x are interpreted as messages (beliefs) living 
on the edges of the factor graph, with the consistency rules (jl.l6a| and (11.16bl) on the 
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function and variable nodes. Equations (11.161) are usually solved by iteration, the name 
message passing is used in this context. In the following it will be simpler not to consider 
the "two-levels" equations f 1 1 . 1 6 j) but 

€r=z^ e M*sh*i>fl n n (i.i7) 

{sj},j£da—i j£da—ib£dj—a 

where Z^ 1 = Z a ^ % \\- &da _ i Z^ a . Notice that on simple graphs, i.e., when either h = 2 
for all % = 1, . . . , N or k a = 2 for all a = 1, . . . , M, the form (11.171) simplifies further. 
And on constraint satisfaction problems on simple graphs (e.g. the matching or coloring 
problems) the "two-levels" equations are almost never used. 

Assuming that one has found the fixed point of the belief propagation equations 
( ll.16alfl~.16bl) . one can deduce the various marginal probabilities and the free energy, 
entropy etc. The marginal probability HI . 141) of variable i estimated by the BP equations 
is 

a£di 

To compute the free energy we first define the free energy shift AF a+9a after addition 
of a function node a and all the variables i around it, and the free energy shift AF l after 
addition of a variable i. These are given in general by: 

e- pAFa+aa = z^= j2 n n ( li9& ) 

{si],i&da iedab£di—a 

e~^ = Z-^HC- (l-19b) 

Si addi 

The total free energy is then obtained by summing over all constraints and subtracting 
the terms counted twice |MP01LlYFW03] : 

a i 

This form of the free energy is variational, i.e., the derivatives 9 ^fjff)) and 

vanish if and only if the probabilities x l ^ a an d ip a ^ 1 satisfy (ll.16atil.16bp . This allows to 

compute easily the internal energy as 

EG ^>~ dd Z a+da ■ ( L21 ) 

a 

The entropy is then obtained as 

S G ((3) = (3[E G ((3)-F G (P)]. (1.22) 

All the equations fll.16D - fll.22p are exact if the graph G is a tree. The replica symmetric 
approach consists in assuming that all correlations decay fast enough that application of 
eqs. (jl.16p - fll.22p on a large tree-like graph G gives asymptotically exact results. These 
equations can be used either on a given graph G or to compute the average over the 
graph (and disorder) ensemble. 
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1.5.3 Average over the graph ensemble 

We now study the typical instances in an ensemble of graphs. We denote the average 
over the ensemble by E(-). We assume that the random factor-graph ensemble is given by 
a prescribed degree distribution Q(l) for variables and TZ(k) for constraints. Let us call 
V(ip) and 0(x) the distributions of messages ip and x over au the edges of a large typical 
factor graph from the ensemble. They satisfy the following self-consistent equations 

oo „ I 

P(V0=£ff(O R^Otf)] Sty (1.23a) 

1=1 J i=l 

oo „ k 

o( X ) = Y, r ( k ) j n^wi^^-^m)], (i.23b) 



k=l " i=l 



where the functions and T x represent the BP equations fll.16atil.16bp . q(l) and r(k) 



are the excess degree distributions defined in (jl.8p . If there is a disorder in the interaction 
terms, as e.g. the negations in i-T-SAT, we average over it at the same place as over the 
fluctuating degree. 

Solving equations fll.23atil.23bj) to obtain the distributions V and O is not straight- 



forward. In some cases (on regular factor graphs, at zero temperature, etc.) it can be 
argued that the distributions V, O are sums of Dirac delta functions. Then the solution 
of eqs. f ll.23atil~23bl) can be obtained analytically. But in general distributional equa- 
tions of this type are not solvable analytically. However, a numerical technique called 
population dynamics |MP01] is very efficient for their resolution. In appendix [E] we give 
a pseudo-code describing how the population dynamics technique works. 

Once the distributions V and O are known the average of the free energy density can 
be computed by averaging f ll.20p over V . This average expression for the free energy is 
again in its variational form (see |MP01j ). i.e., the functional derivative ^ffi vanishes if 
and only if V satisfies (11.321) . The average energy and entropy density are thus expressed 
again via the partial derivatives. 



Factorized solution — As we mentioned, on the ensemble of random regular factor 
graphs (without disorder in the interactions) the solution of equations ( 11.231) is very 
simple: V(ip) = 5(ip — ip ms ), Q{x) — S(x ~ X rcg )i where ip reg and x reg is a self-consistent 
solution of (11.161) . This is because in the thermodynamical limit an infinite neighbourhood 
of every variable is exactly identical thus also the marginal probabilities have to be 
identical in every physical solution. 



1.5.4 Application for counting matchings 

To demonstrate how the replica symmetric method works to compute the entropy, that 
is the logarithm of the number of solutions, we review the results for matching on sparse 
random graphs [ZDEB-1]. The reasoning why the replica symmetric solution is exact for 
the matching problem is done on the level of self-consistency checks in [ZDEB-1]. And 
[BN06] have worked out a rigorous proof for graphs with bounded degree and a large 
girth (length of the smallest loop). 

Consider a graph G(V, E) with N vertices (N = \V\) and a set of edges E. A matching 
(dimerization) of G is a subset of edges M C E such that each vertex is incident with 
at most one edge in M. In other words the edges in the matching M do not touch each 
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other. The size of the matching, \M\, is the number of edges in M. Our goal is to compute 
the entropy of matchings of a given size on a typical large Erdos-Renyi random graph. 

We describe a matching by the variables Sj = 6 {0, 1} assigned to each edge 
i = (ab) of G, with s i = 1 if % 6 M and Sj = otherwise. The constraints that two edges 
in a matching cannot touch impose that, on each vertex a G V^: ( ^g^ S( a &) < 1. To 
complete our statistical physics description, we define for each given graph G an energy 
(or cost) function which gives, for each matching M = {s}, the number of unmatched 
vertices: 

H G (M = {s}) = J2 E a ({s}) = N-2\M\, (1.24) 



where E a = 1 - J2 db s {ab) . 

In the factor graph representation we transform the graph G into a factor graph F(G) 
as follows (see fig. 11.51) : To each edge of G corresponds a variable node (circle) in F(G); 
to each vertex of G corresponds a function node (square) in F(G). We shall index the 
variable nodes by indices i, j, k, . . . and function nodes by a, b, c, ... . The variable i takes 
value Si = 1 if the corresponding edge is in the matching, and = if it is not. The 
weight of a function node a is 



b a ({da},(3) =1 ( J2 s i < 1 ) e-^-Eiea.*) 

\i&da J 



(1.25) 



where da is the set of all the variable nodes which are neighbours of the function node a, 
and the total Boltzmann weight of a configuration is z^m Yl a ^adda}, j3). 





Figure 1.5: On the left, example of a graph with six nodes and six edges. On the right, 
the corresponding factor graph with six function nodes (squares) and six variable nodes 
(circles). 



The belief propagation equation f 1 1 . 1 6 1) becomes 



{ Sj } \ jedb- 



Sj<l\ e-ft 1 -*-^ 



II x 

j&db—i 



j^b 

Si i 



1.26) 



where Z b ^ a is a normalization constant. In statistical physics the more common form of 
the BP equations uses analog of local magnetic fields instead of probabilities. For every 
edge between a variable i and a function node a, we define a cavity field h l ^ a as 



^.i — >a 



The recursion relation between cavity fields is then: 



h^ a = --lo§ 

P 



+ E 



jGdb—i 



1.27) 



1.28) 
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The expectation value (with respect to the Boltzmann distribution) of the occupation 
number Si of a given edge i = (ab) is equal to 

1 



I I e -/3(h i ^ a +h i ^ b ) 

The free energy shifts needed to compute the total free energy fll.20[) are 

— f3AF a+ie g a 



-f5AFi 



e -0 + J2e? hi 

■ i— >a i ui— tb\ 



The energy, related to the size of the matching via (11.241) . is then 



D 1 + Z^ie9a e 



This is the sum of the probabilities that node a is not matched. 
The distributional equation (11.231) becomes 



oo „ k 

o{h) = Y, r ( k ) / n i dhi °( hi )] 6 

k=l i=l 

And the average free energy is explicitly 



h + g log | ( 



N 



\ E / II [WOffl] log e"' 3 + e?* 

P k=0 J i=l \ i 



+ ^pj dhl dh2 °( hl ) °( h2 ) lo s ( x + e 



(1.29) 

(1.30a) 
;i.30b) 

(1.31) 



1.32) 



1.33) 



Where TZ(k) is the connectivity distribution of the function nodes, that is the connec- 
tivity distribution of the original graph, c is the average connectivity. The distributional 
equations are solved via the population dynamics method, see appendix [El Fig. I1.6l then 
presents the resulting average entropy as a function of size of the matching. 



1.6 Clustering and Survey propagation 

As we said previously in the random .fT-SAT the replica symmetric solution is not gener- 
ically correct. Mezard and Parisi [MP01J understood how to deal properly and without 
approximations with the replica symmetry breaking on random sparse graphs, that is 
how to take into account the correlations induced by long loops. More precisely in their 
approach only the one-step (at most two-step on the regular graphs) replica symmetry 
breaking solution is numerically feasible. Anyhow, such a progress opened the door to a 
better understanding of the optimization problems on sparse graphs. The ^-satisfiability 
played again the prominent role. 

To compute the ground state energy within the 1RSB approach we can restrict only to 
energetic considerations as described in [MP03J, we call this approach the energetic zero 
temperature limit. Applying this method to /^-satisfiability leads to several outstand- 
ing results [MPZ02, MZ02J, we describe the three most remarkable ones. Soon after, 
analog results were obtained for many other optimization problems, for example graph 
coloring |MPWZ021 lBMP+031 IKPW04] . vertex cover [Zho03j . bicoloring of hyper-graphs 



[CNRTZ03] . XOR-SAT [FLRTZ01L IMRTZ03] or lattice glass models |BM021 [RBMM 04J. 
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Figure 1.6: Entropy density s(m) as a function of relative size of the matching m = \M\/N 
for Erdos-Renyi random graphs with mean degrees c = 1, 2, 3, 6. The lower curve is the 
ground state entropy density for all mean degrees. The curves are obtained by solving 
eqs. fll.32l) -( fL33l) with a population dynamics, using a population of sizes N = 2 ■ 10 4 to 
2 ■ 10 5 and the number of iterations T = 10000. 



Clustering — It was known already in the "pre-lRSB-cavity era" that replica sym- 
metry broken solution is needed to solve random K-SAT. Such a need is interpreted 
as the existence of many metastable well-separated states, in the case of highly de- 
generate ground state this leads to a clustering of solutions in the satisfiable phase 
[BMWOOl IMPZ021 IMZ02] . The energetic 1RSB cavity method deals with clusters con- 
taining frozen variables (clusters with backbones), that is variables which have the same 
value in all the solutions in the cluster. It predicts how many of such clusters exist at 
a given energy, the logarithm of this number divided by the system size N defines the 
complexity function According to the energetic cavity method for 3-SAT, clusters 

exist, S(0) ^ 0, for constraint density a > a SP = 3.92 |MPZ02| IMZ02] . 

It was conjectured [MPZ021 IMZ02| that there is a link between clustering, ergodicity 
breaking, existence of many metastable states and the difficulty of finding a ground state 
via local algorithms. The critical value «sp was called the dynamical transition and the 
region of a > «sp the hard-SAT phase. 

Clusters were viewed as a kind of pure states, however, in the view of many a good 
formal definition was missing. It was also often referred to some sort of geometrical 
separation between different clusters. A particularly popular one is the following: Clusters 
are connected components in the graph where solutions are the nodes and two solutions 
are adjacent if they differ in only d variables. Depending on the model and author the 
value of d is either one of d is a finite number of d is said to be any sub-extensive number. 
The notion of x-satisfiability, the existence of pairs of solutions at a distance x, leads 
to a rigorous proof of existence of exponentially many geometrically separated clusters 
[MMZ051 IDMMZ081 IART06] . 
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The satisfiability threshold computed — The energetic 1RSB cavity method al- 
lows to compute the ground state energy and thus also the satisfiability threshold a s . In 
3-SAT its value is a s = 4.2667 [MPZ021 IMZ021 IMMZ06] . This value is computed as a 
solution of a closed distributional equation. This time there is an excellent agreement 
with the empirical estimations. Is the one step of replica symmetry breaking sufficient 
to locate exactly the satisfiability threshold? The stability of the 1RSB solution was 
investigated in [MPRT04J, the 1RSB energetic cavity was shown to describe correctly the 
ground state energy for 4.15 < a < 4.39 in 3-SAT. In particular, it yields the conjecture 
that the location of the satisfiability threshold is actually exact. From a rigorous point 
of view it was proven that the 1RSB equations give an upper bound on the satisfiability 
threshold [FL031 IFTTM IPTTM] . 

Survey Propagation: a revolutionary algorithm — The most spectacular result 
was the development of a new message passing algorithm, the survey propagation [MZ02, 
BMZ05J. Before the replica and cavity analysis were used to compute the quenched 
averages of thermodynamical quantities. Using always the self-averaging property that 
the average of certain (not all) quantities is equal to their value on a large given sample. 
Mezard and Zecchina applied the energetic 1RSB cavity equations, later called survey 
propagation, on a single large graph. This resulted in an algorithm which is arguably still 
the best known for large instances of random 3-SAT near to the satisfiability threshold. 
And even more interesting than its performance is the conceptual advance this brought 
into applications of statistical physics to optimization problems. 



In this section we derive the energetic zero-temperature limit of the 1RSB method. When 
applied to the satisfiability problem this leads, between others, to the calculation of 
the satisfiability threshold and to the survey propagation equations and algorithm. We 
illustrate this on the l-in-3 SAT problem. Before doing so we have to introduce the 
warning propagation equations, on which the derivation of the survey propagation relies. 

1.7.1 Warning Propagation 

In general warning propagation (min-sum) is a zero temperature, (3 — > oo, limit of the 
belief propagation (sum-product) equations fll.16aHl.16bj) . It can be used to compute the 
ground state energy (minimal fraction of violated constraints) at the replica symmetric 
level. A constraint satisfaction problem at a finite temperature gives rise to <j) a ({da}, (3) = 
1 if the constraint a is satisfied by configuration {sg a }, and <f) a ({da}, (3) = e _2/3 if a is 
not satisfied by {sa a }0. In a general Boolean CSP, with N variables G { — 1,1}, the 
warning propagation can then be obtained from fll.16am.16bj) by introducing warnings u 
and h as 



1.7 Energetic 1RSB solution 





2 The factor 2 in the Hamiltonian is introduced for convenience and in agreement with the notation 
of [ZDEB-3]. 
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This leads in the limit of zero temperature, (3 — > oo, to 

h ir+a = u h ^\ (1.35a) 

badi—a 

^ ^%-2^({ Sj },+i)) 

i68a_< (1.35b) 
-max( V h?-* a 8 j -2E a {{s j },-\)) . 

{Sj} 

jdda—i 

where E a ({si}) = if the configuration {s^ satisfies the constraint a, and E a ({si}) = 1 if 
it does not. The warnings u and h have to be integer numbers, as they can be interpreted 
as changes in the ground state energy of the cavity subgraphs when the value of variable 
i is changed from s, = to = 1. Given E a G {0, 1} we have that h G Z and 
u G { — 1, 0, +1}. The correspondence between values of u and ip are 

u = 1 ^ ^i = l, ^-i=0, (1.36a) 
it = — 1 <J=> ^i = 0, ^-i = l, (1.36b) 
w = ^ ^i = e, if>-i = l-e, 0<e<l. (1.36c) 

The warnings u and h can thus be interpreted in the following way 



a 



u 



= -1 

u a ^ { = 



Constraint a tells to variable v. "I think you should be —1." 
Constraint a tells to variable v. "I can deal with any value you take." 
Constraint a tells to variable v. "I think you should be +1." 



ti^ a < 
ti^ a > 



Variable i tells to constraint a: "I would prefer to be —1." 

Variable % tells to constraint a: "I don't have any strong preferences." 

Variable i tells to constraint a: "I would prefer to be +1." 



Given this interpretation the prescriptions (11.35)) on how to update the warnings over the 
graph becomes intuitive. Variable i collects the preferences from all constraints except a 
and sends the result to a. Constraint a then decides which value i should take given the 
preferences of all its other neighbours. 





h 2^a 




+ 


+ 





+ 






+ 




















+ 











Table 1.1: Example of the update ( 11.35b[) in the positive l-in-3 SAT problem, where 
exactly one variable in the constraint takes value 1 in order to satisfy the constraint. The 
first line might seem counter-intuitive, but note that we defined the energy in such a way 
that configuration (1, 1, 1) is as bad as (1, 1, —1). 
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Given the fixed point of the warning propagation (11.351) the total warning of variable 

i is 

^ = ^V^\ (1.37) 

addi 

The corresponding energy can be computed as 

E = J2^E a+9a -^(/;-l)A£\ (1.38) 

a i 

where AE a+9a is the number of contradictions created when constraint a and all its 
neighbours are added to the graph, AE" 1 is the number of contradictions created when 
variables % is added to the graph. The energy shifts can be computed from (ll.19atil.19bj) 
using (ll.34p and taking (3 — > oo they read 



AE a+da = -m&x[J2 hi ~* as i- E »({ s da})} + J2 E i^i; ( L39a ) 
AE ' 1 = "1 + Yl \ uarM \ ; ( L39b ) 



a&di aGdi 



To summarize, the warning propagation equations neglect every entropic information 
in the belief propagation (ll.16atil.16bj) . thus only the ground state energy can be com- 



puted. On the other hand the fact that warnings u and h have a discrete set of possible 
values simplifies considerably the average over the graph ensemble presented in sec. 11.5.31 
as the distribution V is a sum of three Dirac function, and can be represented by their 
weights. Deeper interpretations of warning propagation and its fixed points will be given 
in chapter HI Note that in the literature the value of warnings is also called * or "joker" 
|BMWZ03jlBZ04] . 



1.7.2 Survey Propagation 

Survey propagation (SP) [MPZ02j IMZ02] is a form of belief propagation which aims to 
count the logarithm of the number of fixed points of warning propagation (11.351) of a 
given energy fll.38j) . For the sake of simplicity we present the most basic form of SP 
which aims to count the logarithm of number of fixed points of the warning propagation 
with zero energy. 

The constraints on values of the warnings assuring that the fixed point of warning 
propagation corresponds to zero energy are 

• For all i and a G di: the warnings {u 6 ^*}b e< 9i_ a are all non-negative or all non- 
positive, 

• For all a and % G da: the preferred values of all j G da — i can be realized without 
violating the constraint a. 

We define probabilities that warnings u a ~* % or h l ^ a are positive, negative or null. 

V a ^\u a ^) = tf^Sivr* + 1) + q^Siu^ - 1) + q^5(u a ^) ; (1.40a) 
V i ^ a (h i ^ a ) = P r>_(/i^ a ) + p^ a /i + (^ a ) + pr a 5(ti-* a ) ; (1.40b) 

where q ^ 1 + q°^ % + q^ 1 = p % Z* a + p!p a + p}^ a = 1, and n±(h) are normalized measures 
with support over ZrK So, to every oriented edge we associate a message q = (g_, go, <?+) 
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or p = (p_,p ,P+) (resp. if oriented towards the variable or the constraint). We call these 
messages surveys, they are analogous to beliefs ip a ~*' 1 and x t ^ a from fll.16atll.16bl) . And 
thus, if the factor graph is tree, exact iterative equations for q, p can be written. The 
update of surveys p given incoming q is common for all Boolean CSPs and reads: 

pv a + vr a = n far* + > ( L41a ) 

badi—a 

v l r a + pr a = K\ n faT* + ^) > ( L41b ) 

badi—a 

pr a =K\ n sT*. ( L41c ) 

badi—a 

where M^ a is the normalization factor. The update of surveys q given the incoming 
ps depends on the details on the constraint functions. For concreteness we write the 
equation for the positive l-in-3 SAT problem. The constraints assuring zero energy then 
forbids that both the warnings incoming to a constraint a have value +1. 

q"p i =M^ i p j r a p k r a , (1.42a) 
= jv-i. [^-a (l _ ^ tt) + (1 _ pH*)jfc~] f ( L 42b) 

= K\ [^ a vl^ a + ^ a p k r a + pT>r°] , (1.42c) 

where A/" a _»i = 1 — p 1 ^ a p k ^ a is the normalization factor, j and are the other two 
neighbours of a. 

The associated Shannon entropy is called complexity [Pal83j (or structural entropy in 
the context of glasses) and reads |MZ02j 

E{E = 0) = J2 lQ g M a+9a - Y^{U ~ l ) lo S^ > ( L43 ) 

a % 

where J\f a+da is the probability that no contradiction is created when the constraint a 
and all its neighbours are added, M l is the probability that no contradiction is created 
when the variable % is added. Remark the exact analogy with f ll,19atfl~19b[) . We denote 
H = U a edi <fiT and V' ± = HaedM^ + 9^). then 

M l = V\+V i _-Vl, (1.44a) 
„ (1.44b) 

iGfla jSida—i 

The second equation collects the contributions from all combinations of arriving sur- 
veys except the "contradictory" ones (+,+,+), (—,—,—), (+,+,0) and (+,+,—) (plus 
permutations of the latter). 

The survey propagation equations fll.4mi.42j) and the expression for the complexity 
function (j 1.43 ft are exact on tree graphs. In the spirit of the Bethe approximation, we 
will assume sufficient decay of correlations and use these equations on a random graph H. 
To average over the ensemble of random graphs we adopt the same equations as we did 
for the belief propagation in sec. 11.5.31 



3 The fact that on a given tree with given boundary conditions the warning propagation has a unique 
fixed point might seem puzzling at this point. Clarification will be made in the chapter [2l 



22 



CHAPTER 1. HARD OPTIMIZATION PROBLEMS 



1.7.3 Application to the exact cover (positive l-in-3 SAT) 

The l-in-3 SAT problem (with probability of negating a variable equal to one-half) is 
a rare example of an NP-complete problem which is on average algorithmically easy 
and where the threshold can be computed rigorously [ACIM01J. In particular it was 
shown that for a ^ 1 an instance of the problem can be solved in polynomial time with 
probability going to one as N — > oo. This result was generalized into random l-in-3 SAT 
where the probability of negating a variable isp ^ 1/2 [ZDEB-3]. In particular we showed 
that for all 0.273 < p < 0.718 the RS solution is correct and almost every instance can 
be solved in polynomial time if the constraint density a ^ l/[4p(l — p)]. When, however, 
p < 0.273 the phase diagram is more complicated, see [ZDEB-3]. For p = the solution of 
the positive l-in-3 SAT (exact cover) problem becomes very similar to the one of 3-SAT 
[MZ02J. The result for the complexity (11.431) in the positive l-in-3 SAT obtained from 
the population dynamics method is plotted in fig. 11.71 For more detailed discussion of 
how the phase diagram changes from the almost-always-easy to the very-hard pattern see 
[ZDEB-3]. 
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Figure 1.7: Average complexity density (logarithm of number of states divided by the 
number of variables) as a function of the mean degree c for the positive l-in-3 SAT prob- 
lem. At csp = 1.822 a nontrivial solution of the survey propagation equations appears, 
with positive complexity. At c s = 1.8789 ± 0.0002 the complexity becomes negative: this 
is the satisfiability transition. At c p = 1.992 the solution at zero energy ceases to exist. 
The inset magnifies the region where the complexity crosses zero, together with the error 
bar for the satisfiability transition. Crosses represent results of a population dynamics 
with N = 0.5 ■ 10 5 elements, squares of AT = 1 • 10 5 , and circles N = 2 ■ 10 5 . 



Up to certain average connectivity of variables csp = 1.822 the only iterative fixed 
point of the population dynamics gives q^ 1 = p\^ a = 1 for all (ia). The associated 
complexity function is zero. In an interval (csp,c s ) = (1.822,1.879) there exist a non- 
trivial solution giving positive complexity function. There are thus exponentially many 
different fixed points of the warning propagation. Asymptotically, almost every warning 
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propagation fixed point is associated to a cluster of solution^. Above c s = 1.879 there 
is a nontrivial solution to the SP equations giving a negative complexity function. There 
are thus almost surely no nontrivial fixed points of warning propagation at zero energy. 

Before interpreting the survey propagation results, we should check that its application 
on tree-like random graphs is justified. The method to do this self-consistency check has 
been developed in |MPRT04] and is discussed in appendix [TJ For l-in-3 SAT the result in 
that SP is stable, thus the results are believed to be correct, for c G (1.838, 1.948) [ZDEB- 
3]. The point c s belongs to this interval, thus we can interpret it safely as the satisfiability 
threshold. However, the point csp has no physical meaning, and some statements that 
are suggested by its existence are wrong. For example it is not true that there is not 
exponentially many fixed points of the warning propagation, thus no clustering, for c < 
csp. This has been remarked in [ZDEB-4] and a part of chapter [2] will be devoted to 
understanding this. 

1.8 Loose ends 

We could summarize the understanding of the subject three years ago in the following way: 
The 1RSB cavity method was able to compute the satisfiability threshold. The clustered 
phase was predicted and its existence partially proven. The conjecture that clustering 
is a key element in understanding of the computational hardness was accepted. The 
survey propagation inspired decimation algorithm was breath-taking, and the computer 
science community was getting gradually more and more interested in the concepts which 
lead to its derivation. It might have seemed that a real progress can be made only on 
the mathematical side of the theory, in the analytical analysis of the performance of the 
message passing algorithms, or in new applications. But several loose ends hanged in the 
air and the opinions on their resolution were diverse. I will list three of them which I 
consider to be the most obtruding ones. 

(A) The "no man's land", RS unstable but SP trivial — The energetic 1RSB 
cavity method (survey propagation) predicts the clustering in 3-SAT at «sp = 3.92. But 
the replica symmetric solution is unstable at already a^s — 3.86, at this point the spin 
glass susceptibility diverges and equivalently the belief propagation algorithm stops to 
converge on a single graph, see appendix O What is the solution in the "no man's 
land" between ckrs and asp? The values are even more significant for the 3-coloring 
or Erdos-Renyi graphs where the corresponding average connectivities are crs = 4 and 
c S p = 4.42. 

(B) No solutions with nontrivial whitening cores — An iterative procedure called 
whitening of a solution is defined as iteration of the warning propagation equations ini- 
tialized from a solution. Whitening core is the corresponding fixed point. We call white 
those variables which are assigned the "I do not care" state in the whitening core. A 
crucial asymptotic property is that if the 1RSB solution is correct then the whitening 
core of all solutions from one cluster is the same and the non-white variables are the 

4 There might exist fixed points of the warning propagation which are not compatible with any solution, 
thus do not correspond to a cluster. Such "fake" fixed points are negligible if the 1RSB approach is 
correct. 
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frozen ones in that cluster. Consequently, knowing a solution, the whitening may be used 
to tell if the solution was or was not in a frozen cluster. 

Survey propagation uses information only about frozen cluster. It might seem that 
every cluster is uniquely described by its whitening core, that is by the set and values of 
the frozen variables. 

Yet, the solutions found by survey propagation have always a trivial, all white, whiten- 
ing core. This paradox was pointed out in |MMW07] and observed also by the authors 
of |BZ04j . It was suggested that the concept of whitening might be meaningful only in 
the thermodynamical limit. But that was not a satisfactory explanation. 

(C) Where do the simple local algorithms actually fail — The clustered phase, 
baptized "Hard" in [ MZ02] does not seem to be that hard. There is no local algorithm 
which would perform well exactly up to asp = 3.92. For a while it was thought that the 
1RSB stability point ajj = 4.15, see appendix [D] , is a better alternative. It was argued 
that the full-RSB states are more "transparent" for the dynamics than the 1RSB states 
which should be well defined and separated. Moreover there was at least one empirical 
result which suggested that the Walk- SAT algorithm stops to work in linear time at 
that point |AGKQ4j . But other version of Walk-SAT stopped before or even after, as for 
example the ASAT which was argued in [AA06J to work in linear time at least up to 
a = 4.21. 

1.9 Summary of my contributions to the field 

In my first works [ZDEB-1, ZDEB-2, ZDEB-3] I applied the replica symmetric and the 
energetic 1RSB method to the matching and the l-in-K SAT problems. This is why I 
used these two problems to illustrate the methods in sec. 11.5.41 and 11.71 

The problem of matching on graphs is a common playground for algorithmic and 
methodological development. I studied the problem of counting maximum matchings in 
a random graph in [ZDEB-1]. Finding a maximum matching is a well known polynomial 
problem, while their approximative counting is a much more difficult task. We showed, 
that the entropy of maximum matchings can be computed using the belief propagation 
algorithm, a result which was later on partially proved rigorously |BN06j . 

My interest in the l-in-.fr SAT problem stemmed from the work ACIMOl] where the 
authors computed rigorously the satisfiability threshold and showed that the NP-complete 
problem is in fact on average algorithmically easy. In [ZDEB-2, ZDEB-3] we studied the 
random l-in-3 SAT in two-parameter space. One parameter is the classical constraint 
density, the other is the probability p of negating a variable in a constraint (p = 1/2 in 
|ACIM0l~] ). We showed that for 0.2627 < p < 0.7373 the problem is on average easy and 
the satisfiability threshold can be computed rigorously. On the other hand for p < 0.07 
the problem is qualitatively similar to the 3-SAT. We computed the threshold from the 
energetic 1RSB approach. In the intermediate region the 1RSB approach is not stable, 
thus it stays an open question how exactly does the problem evolve from an on average 
easy case to a 3-SAT like case. Qualitatively similar phase diagram was described in 
the 2 + p SAT problem [MZK+99a[ lAKKKOlj . We also found an interesting region of 
the parameter space in the l-in-3 SAT where the unit clause algorithm provably finds 
solutions despite the replica symmetric solution being not correct (unstable). 

The rest of my works [ZDEB-4, ZDEB-5, ZDEB-6, ZDEB-7, ZDEB-8, ZDEB-10, 
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ZDEB-9] tied up the loose ends from the previous section and mainly addressed the orig- 
inal question of this thesis: Why are some constraint satisfaction problems intrinsically 
hard on average and what causes this hardness? 

I used the entropic zero temperature 1RSB approach, introduced in [MPR05J, to 
study the structure of solutions in random CSPs. In [ZDEB-4, ZDEB-5] we discovered 
that the true clustering (dynamical) transition does not correspond to the onset of a 
nontrivial solution of the survey propagation equations. We gave a proper definition of 
the clustering transition and formulated it in terms of extremality of the uniform measure 
over solutions. The clustering transition happens always before or at the same time as 
the replica symmetric solution ceases to be stable. This tied up the loose end (A), as in 
the "no man's land" the energetic 1RSB solution was simply incomplete. 

We showed that in general there exist two distinct clustered phases below the satis- 
fiable threshold. In the first, dynamic clustered phase, an exponentially large number of 
pure states is needed to cover almost all solutions. However, average properties (such as 
total entropy) still behave as if the splitting of the measure did not count. In particular, 
a simple algorithm such as belief propagation gives asymptotically correct estimates of 
the marginal probabilities. However, the measure over solutions is not extremal and, 
more importantly, the Monte Carlo equilibration time diverges, thus making the sam- 
pling of solutions a hard problem. The second kind of clustered phase is the condensed 
clustered phase where a finite number of pure states is sufficient to cover almost all so- 
lutions. A number of nontrivial predictions follows: for instance the total entropy has 
a non-analyticity at the transition to this phase, the marginal probabilities are non-self- 
averaging and not given anymore by the belief propagation algorithm. 

In the context of the coloring problem, i.e. anti- ferromagnetic Potts glass, I also 
addressed related questions of what does the 1RSB solution predict for the finite tem- 
perature phase diagram and when is the 1RSB solutions correct (stable) [ZDEB-5]. We 
give the full phase diagram for this model and argue that in the colorable phase for at 
least 4 colors the 1RSB solutions is stable, and thus believed to be exact. 

In order to clarify and substantiate this heuristic picture, we introduced the random 
subcubes model in [ZDEB-8], a generalization of the random energy model. The random 
subcubes model is exactly solvable and reproduces the sequence of phase transitions 
in the real CSPs (clustering, condensation, satisfiability threshold). Its, perhaps, most 
remarkable property is that it reproduces quantitatively the behaviour of random q- 
coloring and random i^-SAT in the limit of large q and K. We showed that the random 
subcubes model can also be used as a simple playground for the studies of dynamics in 
glassy systems. 

An important and quite novel phenomena I investigated in [ZDEB-5, ZDEB-7] is the 
freezing of variables. A variable is frozen when in all the solutions belonging to one cluster 
it takes the same value. I discovered that the fraction of such frozen variables undergoes 
a first order phase transition when the size of states is varied. I introduced the notion 
of the rigidity transition as the point where almost all the dominating clusters become 
frozen and the freezing transition as the point where all the clusters become frozen. The 
solutions belonging to the frozen clusters can be recognized via the whitening procedure. 

We computed the rigidity transition in the random coloring in [ZDEB-5]. And we 
studied the freezing transition in 3-SAT numerically [ZDEB-10], with the result aif = 
4.254 ± 0.009 (to be compared to the satisfiability threshold a s = 4.267). This study 
also confirms that the notion of whitening and freezing of variables in meaningful even 
on relatively small systems. 
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This allows us to tie up the loose end (B). The survey propagation algorithm describes 
the most numerous frozen clusters. The range of connectivities where the SP based 
algorithms are able to find solutions in 3-SAT lies in the phase where most solutions are 
in fact unfrozen. It is thus much less surprising that the SP based algorithms always find 
a solution with a trivial whitening. 

A very natural question cannot be avoided at this point: What happens in the frozen 
phase where all the solutions are frozen? We know that such a phase exists, this was 
shown in [ART06] and numerically in [ZDEB-10]. And we also know from several authors 
that the known algorithms do not seem to be able to find frozen solutions in polynomial 
time (that is never for sufficiently large instances). We conjectured in [ZDEB-5] that the 
freezing is actually a relevant concept for the algorithmical hardness. Thus the answer 
we suggest to tie up the loose end (C) is that the simple local algorithms stop always 
before the freezing transition. It is a challenging problem to design an algorithm which 
would be able to beat this threshold. 

In the coloring and satisfiability problems (at reasonably small q and K) the freezing 
transition is however very near to the satisfiability threshold, see the numbers in [ZDEB- 
5, ZDEB-10]. It is thus difficult to make strong empirical conclusions about the relation 
between hardness and freezing. Motivated by the need of problems where the freezing 
and satisfiability would be well separated I introduced the locked constraint satisfaction 
problems where the freezing transition coincides with the clustering one [ZDEB-9]. The 
locked CSPs are very interesting from several points of view. The clusters in locked CSPs 
are point-like, this is why the clustering and freezing coincide. This is also connected 
with a remarkable technical simplification, as these problems can be fully described on 
the replica symmetric level. 

On the other hand the locked problems are extremely algorithmically challenging. We 
implemented the best known solvers and showed that they do not find solutions starting 
very precisely from the clustering (= freezing) transition. At the same time this transition 
is very well separated from the satisfiability threshold. 

A remarkable point about a subclass of the locked problems which we called balanced is 
that the satisfiability threshold can be obtained exactly from the first and second moment 
calculation. This adds a huge class of constraint satisfaction problems to a handful of 
other NP-complete CSPs where the threshold is known rigorously. And it also brings the 
understanding of which properties of the problem introduce fluctuations which make the 
second moment method fail. 

The numerical work on the 3-SAT problems [ZDEB-10] also addresses another im- 
portant and almost untouched question: How much are the asymptotic results relevant 
for systems of practical sizes. We counted the number of clusters in random 3-SAT on 
instances up to size N = 150 and compared to the analytical prediction. We saw that 
the comparison is strikingly good for already so small systems. This should encourage 
the application of statistical physics methods to the real world problems. 
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In this chapter we introduce the concept of clustering of solutions. First we investigate 
when does the replica symmetric solution fail. Then we derive the one-step replica sym- 
metry breaking equations on trees and give their interpretation on random graphs. We 
discuss how several geometrical definitions of clusters might be related to the pure states 
and review the properties of the clustered phase. Finally, we revise how is the clustering 
related to the algorithmical hardness and conclude that it is considerably less than previ- 
ously anticipated. The original contributions to this chapter were published in [ZDEB-4, 
ZDEB-5, ZDEB-10]. 

2.1 Definition of clustering and the 1RSB approach 

How to recognize when is the replica symmetric solution correct? First we have to explain 
what do we precisely mean by "being correct". We obviously require that quantities like 
the free energy, energy, entropy, marginal probabilities (magnetizations) are asymptoti- 
cally exact when computed in the replica symmetric approach. But this is not enough, as 
this is also satisfied in the phase which we will call later the clustered (dynamical) 1RSB 
phase. 

A commonly used necessary condition for the validity of the RS solution is referred to 
as the local stability towards 1RSB. It consists in checking that the spin glass susceptibil- 
ity does not diverge, or equivalently that the belief propagation algorithm converges on 
a large single graph, or in the probability theory this corresponds to the Kesten-Stigum 
condition [KS66a, KS66b]. These and other equivalent representations for the replica 
symmetric stability are discussed in detail in appendix If the replica symmetric solu- 
tion is not stable then it predicts wrong free energy, entropy, correlation functions, etc. 
But the contrary is far from being true: even if stable, the RS solution might be wrong, 
and even unphysical (predicting negative entropies in discrete models, negative energies 
in models with strictly non-negative Hamiltonian function, or discontinuities in functions 
which physically have to be Lipschitzian). 

It is tempting to say: The replica symmetric solution is correct if and only if the 
assumptions we used when deriving it are correct. In deriving the belief propagation 
(11.16!) an d the RS free energy (11.201) we used only one assumption: The neighbours of a 
variable i are independent random variables, under the Boltzmann measure (11. 13)) . when 
conditioned on the value of i. As we will see, this assumption is asymptotically correct 
also in the dynamical 1RSB phase, and thus the RS marginal probabilities, or the free 
energy function remain asymptotically exact in that phase. 
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We thus need a different definition for the "RS correctness" which would determine 
whether the Boltzmann measure ( 11.131) can be asymptotically described as a single pure 
state, and whether the equilibration time of a local dynamics is linear in the system 
size. At the same time we do not want this definition to refer the RSB solution, because 
obviously we want to justify the need of the RSB solution by the failure of the RS solution. 

A definition satisfying the above requirements appeared only recently |MM081 IMS05. 
MS06c], and it can be written in several equivalent ways. From now on we say that the 
replica symmetric solution is correct if and only if one of the following is true. 

(a) The point-to-set correlations decay to zero. 

(b) Reconstruction on the underlying graph in not possible. 

(d) The uniform measure over solutions satisfies the extremality condition. 

(c) The 1RSB equations at m = 1, initialized in a completely biased configuration, 
converge to a trivial fixed point. 

In the rest of this section we explain these four statements, and show that they are indeed 
equivalent, and explain how do they correspond to the existence of a nontrivial 1RSB 
solution. We should mention that in the so-called locked constraint satisfaction problems 
this definition have to be slightly changed at zero temperature, we will discuss that in 
sec. 14.31 The transition from a phase where the RS solution is correct to a phase where 
it is not is called the clustering or the dynamical transition. 

Gibbs measures and why are the sparse random graphs different — Our goal 
is to describe the structure of the set of solutions of a constraint satisfaction problem 
with iV variables. Let <fi a (da) be the constraint function depending on variables Sj G da 
involved in the constraint a, (p a (da) = 1 if the constraint is satisfied, <j) a (da) = if not. 
The uniform measure over all solutions can be written as 

1 M 

KM) = zHMfo) , (2.i) 

a=l 

where Z is the total number of solutions. The uniform measure over solutions is the zero 
temperature limit, (3 — > oo, of the Boltzmann measure 

1 M 

The above expressions are valid on any given finite factor graph. The theory of Gibbs 
measures [Geo88| tries to formally define and describe the limiting object to which (12.11 - 
12. 2p converge in the thermodynamical limit, N — ► oo. A common way to build this theory 
is to ask: What is the measure induced in a finite volume A when the boundary conditions 
are fixed? Roughly speaking, the good limiting objects, called the Gibbs measures or the 
pure states, are such that boundaries taken from the Gibbs measure induce the same 
measure inside the finite large volume A. 

The Ising model on a 2D lattice gives an excellent example of how a phase transition 
is seen via Gibbs measures. Whereas in the high temperature paramagnetic phase the 
Gibbs measure is unique, in the ferromagnetic phase there are two extremal measures, one 
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corresponding to the positive average magnetization, the other to the negative average 
magnetization. Indeed, if a boundary condition is chosen from one of these two then the 
correct magnetization will be induced in the bulk. In general the bulk in equilibrium can 
be described by a linear combination of these two extremal objects. 

In the disordered models the situation might be much more complicated. Indeed the 
proper definition of the Gibbs measure in the Edwards- Anderson model (11. lip and other 
glassy models is a widely discussed but still an open problem [Bov06l ITal031 [NS92J. 

The locally tree-like lattices, we are interested in here, are also peculiar from this 
point of view. The main difference is that in any reasonable definition of the boundary 
variables, the boundary has volume comparable to the volume of the interior. Thus 
again the usual theory of Gibbs measure implies very little. On the other hand the tree 
structure makes some considerations simpler. We will try to understand what sort of 
long range correlations might appear on the tree-like graphs by studying the tree graphs 
with general boundary conditions. 



2.1.1 Properties and equations on trees 

It is a well known fact that on arbitrary tree, with arbitrary boundary conditions, the 
belief propagation equations and the Bethe free energy are exact (the thermodynamical 
limit is not even needed here) |Pea88l IKFLOll lYFWOO] . 

But what if the boundary conditions are chosen from a complicated measure? Then 
very little (if anything) is known in general. However, there is a way how to choose the 
boundary conditions such that the tree is then described by the one-step replica symme- 
try breaking equations. This is closely linked to the problem of reconstruction on trees, 
studied in mathematics jEKPSOO, IMosOlj IMo s04j. The link with 1RSB was discovered 
by Mezard and Montanari [MM06aJ. We chose to present the 1RSB equations in this new 
way, because it opens the door to further mathematical developments. For the original 
statistical physics derivation we refer to [MPOOJ. Another recent computer science-like 
derivation, which is based on the construction of a decorated constraint satisfaction prob- 
lem and writing belief propagation on such a problem, in presented in |MM08[ Mor07j. 



Reconstruction on trees — We explain the concept of reconstruction on trees [Mos04] . 
For simplicity we consider g-coloring on a rooted tree with constant branching factor 7 
(sometimes also called the Cayley tree). A more general situation (with disorder, in the 
interaction or in the branching factor) is described in appendix [A] 

Create a rooted tree with branching 7 and with L generations. An example of 7 = 2 
and L = 8 is in fig. 12.11 Assign a color s to the root and broadcast over the edges 
towards the leaves of the tree in such a way that if a parent node i was assigned color Sj 
then each of its ancestors is assigned random one of the remaining q — 1 colors. 

At the end of this broadcasting, every node in the tree is assigned a color, and this 
assignment corresponds to a proper coloring (neighbours have different colors). Now in 
an imaginary experiment we forget the colors everywhere but on the leaves. The problem 
of reconstruction consists in deciding if there is any information left in the values on the 
leaves (and their correlation) about the original color sq of the root in the limit of infinite 
tree L — > 00. If the answer is yes then we say that the reconstruction is possible, if the 
answer is no then the reconstruction is not possible. 

Call {s}i the assignment of colors in the I th generation of the tree. Consider formally 
the probability ip So {{s}i) that a broadcasting process which finished at the configuration 
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Figure 2.1: Illustration of the broadcasting of colors on a binary tree (7 
reconstruction problem. 



2) for the 



{s}i started from the color So at the root. In other words, in what fraction of assignments 
in the interior of the tree (compatible with the boundary conditions {s};) is the color of 
the root s ? Reconstruction is possible if and only if 

q 

lim y> r ({s},)log [qM{s}i)} > 0. (2.3) 

r=l 

Intuitively when the branching 7 is small and the number of colors large the infor- 
mation about the root will be lost very fast. If, on the contrary, the branching is large 
compared to the number of colors some information remains. A simple exercise is to 
analyze the so-called naive reconstruction algorithm |Scm08j. The naive reconstruction 
is possible if the probability that the leaves determine uniquely the root does not go to 
zero as the number of generation goes to infinity. We compute the probability 77 that the 
far-away boundary is compatible with only one value of the root. Denote 7]i the proba- 
bility that a variable in the I th generation is directly implied conditioned on the value of 
its parent. The probability 77;_i can be computed recursively as 

w 1 N7 (q-l)(q-2), 2 N7 

m-i = i-(q-i)(i-^- 1 viY + - — f — 

I- 1 / 1\ 

/v 1 — • ( 2 - 4 ) 

r /V q — 1 / 



B-ir ,-lv ' ' 

r=0 



The terms in this telescopic sum come from probabilities that number r out of the q — 1 
colors are not present in the 7 descendants. In the last generation we know the colors 
by definition of the problem, thus 7]^ = 1. If the iterative fixed point of (12 A\\ is positive 
then the reconstruction is possible. 
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This simple upper bound on the branching 7 for which the reconstruction is possible 
is actually quite nontrivial and in the limit of large number of colors it coincides with 
the true threshold at least in the first two orders, see [ZDEB-5] and |Sem 08, Sly08|. This 
upper bound is connected to the presence of frozen variables and will be discussed in a 
greater detail in chapter HI 



Self-consistent iterative equations for the reconstruction — The iterative equa- 
tions for the reconstruction problem are equivalent to the one-step replica symmetry 
breaking equations with Parisi parameter m, m — 1 will apply to the original question of 
reconstructibility. This was first derived by Mezard and Montanari [MM06aJ and it has 
some deep consequences for the understanding of the RSB solution. We now explain this 
derivation, still for the coloring problem with a fixed branching 7 and q colors. A more 
general form is presented in appendix [Al 

For given boundary conditions {s}i, constructed as described above, we compute the 
probability tp 1 ^ (over all broadcasting experiments leading to these boundary conditions) 
that a variables i had color Sj, where j is the parent of i and the edge (ij) has been cut. 
Given the probabilities on the descendants of i, which are indexed by k = 1, . . . , 7, we 
can write 

v>tr* = z^jU^- O = FsM^}) , (2.5) 
fc=i 

because the descendants can take any other color but Sj. The Z l ^i is a normalization 
constant. It should be noticed that this is in fact the belief propagation equation (jl.lfip for 
the graph coloring. This equation can also be derived by counting how many assignments 
are consistent with the boundary conditions {s}i. This gives a natural interpretation to 

where Z® is the total number of solutions consistent with {s}i if i were the root. Thus 
Z l ^i is a change in the number of solutions compatible with the boundary conditions 
when the 7 branches are merged. 

Now we consider the distribution over all possible boundary conditions which are 
achievable by the broadcasting process defined above. We have to specify the probabil- 
ity distribution on the boundary conditions. We consider that the probability of every 
boundary conditions {s}i is proportional to the power m of the number of ways by which 
we could create {s}/, denote this number Z({s}i). In other words, the probability of 
a given boundary condition is proportional to the power m of the number of possible 
assignments in the bulk of the tree. 



MMO = [Zi z ( } J} , where Z{m) = £ [Z({ a },)] m . (2.7) 

1 ' {*}l 

The value of m = 1 is natural for the original question of reconstruction, because every 
realization of the broadcasting experiment is then counted in a equiprobable way. We, 
however, introduced a general power m. The parameter m will play a role of the Legendre 
parameter, changing its value focuses on boundary conditions compatible with a given 
number of assignments inside the tree. 
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Denote P l ^^(tfj 1 ^^) the distribution of i/> t->J ', over the measure on the boundary con- 
ditions (E2D 



P^(V^) = a (M' ^duce ^ ^^fV • (2-8) 

{s}i 

Where Z^\{s}i) is the number of solutions induced on the subtree rooted in vertex 
i, Z^{m) is the corresponding normalization. To express the probability distribution 
pi-i(y-*i) as a f unc tion of P*-^*"*) we need that ip^ j = ■F({V> fc-4< }), eq. f[2~5l) . 
Moreover, Z l ^i is the increase in the total number of solutions after merging the branches 
rooted at k — 1, . . . , 7 into one branch rooted at i. The distributional equation for P is 
then 

p^i^i) = _L I Y[dP k ^(^) (Z^) m - , (2-9) 

J k=l 

where T and Z l ^i are defined in (12.51) . and Z % ~^i is a normalization constant equal to 

= u l^) {m) = jf[dP k ^ k -) (Z^T ■ (2.10) 

where Z^ is the normalization from (I2.8p if i were the root. Notice that if we start from 
boundary conditions which are not compatible with any solution then the re-weighting 
= at the merging where a contradiction is unavoidable. Initially at the leaves 
the colors of nodes are known. Call 5 r the g-component vector = ( K s i; r )> then the 
initial distribution is just a sum of singletons 

P mit W = -i2 6 ^~ 6 r)- (2.11) 

r=l 

Denote Po(ip) the distribution created from (12.111) after many iteration of (12.91) with 
m = 1. The reconstruction is possible if and only if Po{ip) is nontrivial, that is different 
from singleton on ip s . = 1/q , Vsj. We define the critical branching factor 7^ in such a way 
that for 7 < 7^ the reconstruction is not possible, and for 7 > 7^ the reconstruction is 
possible. The critical values 7^ = q — 1 for the coloring problem are reviewed in tab. 15.21 



What are clusters on a tree? If the reconstruction is not possible, then almost all 
(with respect to (12. 7p at m — 1) boundary conditions do not contain any information 
about the original color of the root. However, for rare boundary conditions this might be 
different. Obviously as long as 7 > q — 1 one can always construct boundary conditions 
which determine uniquely the value of the root (by assigning every of the q — 1 colors to 
the descendants of every node). If 7 < q — 1 then this is no longer possible. And it was 
proven in [Jon02j that for 7 < q — 1 every boundary conditions lead to an expectation 
1/q for every color on the root. If the reconstruction is possible, then different boundary 
conditions may lead to different expectations on the root. 

The basic idea of the definition of clusters on a tree is the same as in the classical 
definition of a Gibbs measure |Geo88j . However, some more work is needed to make the 
following considerations rigorous. Define a ^-neighbourhood of the root as all the nodes 
up to d th generation, consider 1 cZ <d Z. Consider the set S (resp. S') of all assignments 
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on the ^-neighbourhood compatible with a given boundary condition {s}i (resp. {s'}z). 
Define two boundary conditions {s}i and {s'}i as equivalent if the fraction of elements 
in which the two sets S and S' differ goes to zero as I, d — > oo. Clusters are then the 
equivalence classes in the limit I — > oo, d oo, d <ti I. The requirement d <C I comes 
from the fact that in I — d iterations the equation (12. 8p should converge to its iterative 
fixed point. 

As we explained, more than one cluster exists as soon as the branching factor 7 > q—1, 
but as long as the iterative fixed point of eq. (12.91) at m — 1 is trivial all but one clusters are 
negligible because they contain an exponentially small fraction of solutions. Indeed, if the 
reconstruction is not possible it means that the information about the ^-neighbourhood 
is almost surely lost at the I th generation. Thus almost every broadcasting will lead to a 
boundary condition from the only relevant giant cluster. 

Only for 7 > 7^, when the reconstruction start to be possible, the total weight of 
all solutions will be split into many clusters. In every of them the set of expectation 
values (beliefs) ip 1 ^ will be different. This is related to another derivation of the 1RSB 
equations where the clusters of solutions on a given graph are identified with fixed points 
of the belief propagation equations [MM081 IMor07] . There are exponentially many (in 
the total number of variables N) initial conditions, it is also reasonable to expect that 
the number of clusters will be exponentially large in N. 



The complexity function — The number of solutions compatible with a boundary 
condition ({s};) was denoted Z({s}i) in eq. (12.71) . The associated entropy is then, due to 
interpretation of Z 1 ^ (12.61) 

S(M t ) = log [Z({ S },)] =^>gZ^, (2.12) 

i 

where the sum is over all the vertices % in the tree, if % is a leaf then Z % ^i = 1, if % is 
the root that j is a imaginary parent of the root. An intuition about this formula is the 
following: log Z 1 "*^ is the change in the entropy when the node i and all edges (ki), where 
k are descendants of i, are added. Summing over all % then creates the whole tree. 

More commonly, we introduce also messages going from the parents to the descendants 
and write the expression for the entropy (12.121) in the equivalent Bethe form |YFW03j 

sash) = lo s Zl+9t - E lo s zl3 ' ( 2 - 13 ) 



where 



E II - ^ ' zlJ = 1 - E vr'vr* , (2.14) 



7i+di 

r=l k£di 



where di are all the neighbours (descendants and the parent) of node i. The first sum in 
(12.131) goes over all the nodes in the tree, the root included, leaves have only one allowed 
color, thus eq. (12. 14f) changes correspondingly. Again the meaning of log Z i+di is the 
change in the entropy when node i and his neighbourhooding edges are added, each edge 
is then counted twice, thus the shift in the entropy when an edge (ij) is added, \ogZ^ , 
have to be subtracted. 

We denote $(m) = log Z(m) the thermodynamical potential associated to the mea- 
sure (12. 7p . To avoid confusion with the real free energy, associated to the uniform measure 
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over solutions ( 12.11) . we call it the replicated free entropy. If a nonzero temperature is in- 
volved then — $(m)/(/3m) is called the replicated free energy. The replicated free entropy 
on a tree can be expressed in totally analogous way as the entropy. From f 1 2 . 1 j) we derive 

$(m) = log Z[m) =J2 log Z^ j , (2.15) 

i 

which is usually written in the equivalent way 

$(m) = ^logZ i+9i -^log^', (2.16) 



where we introduced 



Z i+di = j JJ dp^^^j^j" (2.17a) 

Z ij = J dP i -* j (i/j i -^ j )dP j -* i (ijj j -' i )(Z ij ) m . (2.17b) 

We denote S(m) the Shannon entropy corresponding to measure on the boundary 
conditions (12. 7p . and we call it the complexity function. 

S(m) = ~J2 MMO lo sMM/) = -mS{m) + $(m) , (2.18) 

{s}i 

where S is the entropy averaged with respect to f^{{s}i) 

S(m) = ^ z{m) IogZ({ a }0 = . (2.19) 

Thus the complexity can also be written as a function of the internal entropy via the 
Legendre transform of the replicated free entropy $(m) 

E(S) = -mS + $(m) with ^3. = _ m . (2.20) 

oS 

The reader familiar with the cavity approach surely recognized eqs. ( 12.91) and ( I2.15l - [2~20l 
as the 1RSB equations. 

Interpretation of the complexity function — In the cavity method [MP01J the 
exponential of the complexity function S(m) ( 12.181) counts the number of clusters cor- 
responding to a given value of the parameter m, that is of a given entropy S ( 12.191) . 
Complexity defined on the full tree is never negative, as it is a Shannon entropy of a 
discrete random variable. The same is, of course true, about the entropy (12 . 1 21) . 

It is more interesting to consider the complexity (or the entropy) function E^(m) on 
the (^-neighbourhood of the root. If the total number of generations of the tree is / we take 
1 < d < L And moreover we require / — d to be large enough, such that the distributional 
iterative equation ( 12.91) converges to its fixed point in less than / — d iterations. The 
average complexity function on the (^-neighbourhood can then be computed from this 
fixed point. And it can be both positive or negative. Its negative value then means that 
the number of clusters is decreasing as we are getting nearer to the root. Two important 
critical connectivities can be defined 
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• 7 C : at which the complexity of the "natural" clusters Ti d (m = 1) becomes negative. 

• 7 S : at which the maximum of the complexity T< d (m = 0) becomes negative. 

The connectivity 7 S is the tree-analog of the satisfiability threshold. The connectivity 7 C 
is the tree-analog of the condensation transition on random graphs, see chapter [3j 

Strictly speaking, it is not known how to justify the interpretation of the complexity 
function as the counter of clusters in the derivation we just presented. In the original 
cavity derivation [MP01J or in the later derivations [MM081 IMor07j this point is well 
justified. We, however, find the purely tree derivation appealing for further progress 
on the mathematical side of the theory and that is why we have chosen to present this 
approach despite this current incompleteness. 

2.1.2 Back to the sparse random graphs 

We stress that the equations, derived in the previous section, are all exact on a given 
(even finite) tree and that we have not use any approximation. We were just describing 
boundary conditions correlated via (12.71) . These, in nature recursive, equations are solved 
via the population dynamics technique, see the appendix [El 

To come back to the sparse random graphs, which are only locally tree-like, we can 
consider equations (l2.9H2.26j ) as an approximation on arbitrary graphs, just as we did 
with belief propagation. This leads to the one-step replica symmetry breaking (1RSB) 
approach. Note that on random graphs we will always speak about densities of the 
entropy, complexity or free-entropy etc. Thus on random graphs: instead of the entropy 
S defined in (" 12. 121) we consider s = S/N. The replicated free entropy $ (12.151) and 
complexity £ (12.181) are also divided by the number of variables. We, however, denote 
them by the same symbol, as confusion is not possible. 

Let us discuss once again, now from the random graph perspective, what are the 
correlations which make the replica symmetric approach fail. This will finally explain the 
definition of the replica symmetric solution being correct given at the beginning of this 
section. 

Point-to-set correlations — The concept of the point-to-set correlations is common 
in the theory of glassy systems. Usually it is considered in the phenomenology of the 
real glassy systems on finite-dimensional lattices, see for example [BB04J and references 
therein. Here we restrict the discussion to properties relevant for the tree-like lattices. 

Call Bd(i) all vertices of the graph which are at distance at least d from i, define 
point-to-set correlation function as 

C d (i) = | \fi,(i, B d (i)) - n(i)n(B d (i))\ | TV , (2.21) 

where /x(-) is the uniform measure over solutions (12. ip . and the total variation distance of 
two probability distributions is defined as \ \q — p||xv = J2 X \q( x ) ~p( x )\/2- The average 
point-to-set correlation is 

1 N 

i=i 

The reconstruction on graphs is then defined via the decay of this correlation function. 
The reconstruction on tree-like graphs is not in general equivalent to the reconstruction 
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on trees. Roughly said, it is not equivalent in the ferromagnetic models, e.g. the ferro- 
magnetic Ising model, which spontaneously break some of the discrete symmetries. On 
the other hand on most of the frustrated models they are equivalent. A general condition, 
which might be very nontrivial to check, is given in [GM07J. 

If the point-to-set correlation function decays to zero, lim^oo Cd = 0, then almost 
every variable is independent of its far away neighbours. The replica symmetric approach 
then has to be asymptotically correct on locally tree-like lattices. 

On the other hand if the point-to-set correlations do not decay to zero, then the 
far-away neighbours influence the value of the variable i. And the replica symmetric 
solution fails to give the correct picture of the properties of the model. The lack of decay 
of the point-to-set correlations is equivalent to the reconstruction on graphs, and is also 
equivalent to the existence of a nontrivial solution of the 1RSB equation (12.91) at m — 1. 
This is also equivalent to the extremality condition for the uniform measure (12.11) . which 
was used in definition of [ZDEB-4] and reads 



where the external average is over quenched disorder (in interactions or connectivities). 

The point-to-set correlations do not decay to zero for example in the low temperature 
phase of the ferromagnetic Ising model on a random graph. There it is sufficient to 
introduce the pure state "up" and the pure state "down" and within these pure states 
the point-to-set correlations will decay to zero again. On the frustrated models the 
situation is more complicated but the idea of the resolution is the same: If we manage 
to split the set of solutions into clusters (pure states) such that within each cluster the 
point-to-set correlations again decay, the situation is fixed. A statistical description of 
the properties of clusters can be obtained using the one-step replica symmetry breaking 
(1RSB) equations, derived in the previous section 12.1.11 and summarized in the next 
section 12.1.31 

However, the correlations might be more complicated and might not be captured 
fully by the 1RSB approach. In particular the 1RSB approach is correct if and only if 
the point-to-set correlation decay to zero within clusters and if the replica symmetric 
statistical description of clusters is correct. In appendix [D] we will discuss a necessary 
condition for the 1RSB approach being correct. In case the 1RSB approach does not fully 
describe the system further steps of replica symmetry breaking might provide a better 
approximation (that means splitting clusters into sub-clusters or aggregation of clusters) 
|MP00j . However, on the tree-like lattices, the exact solutions is not known in such cases. 

Relation with equilibration time — In glasses, the clustering transition is usually 
studied at finite temperature and is called the dynamical transition. The clustered phase 
with S(m = 1) > is called the dynamical 1RSB phase. This phase, where most of 
the static properties do not differ from the replica symmetric (liquid) ones, was first 
described and discuss in |KT87a} IKT87b] . The dynamical transition is associated with a 
critical slowing down of the dynamical properties, e.g. the equilibration time is expected 
to diverge at this point. Note that such a purely dynamical phase transition is typical 
for mean-field models. In the finite dimensional glassy systems the barriers between a 
metastable and an equilibrium state are finite (independent of the system size). This is 
because the nucleation length might be large but have to be finite. Thus instead of a 
sharp dynamical transition in finite dimensional systems we observe only a crossover. 




(2.23) 
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However, even at the mean field level, the exact dynamical description is known only in 
a few toy models, e.g. the spherical p-spin model [CK93] or the random subcubes model 
[ZDEB-8]. In general, the dynamical solution is only approximative, still many very 
interesting results were obtained. For a review see [BCKM98]. In the models on sparse 
random lattices even the approximation schemes are rather poor, see e.g. [SW04j . Thus 
the exact general relation between dynamics and the dynamical (clustering) transition is 
not known. 

An important contribution in establishing the link between dynamics and the static 
solution on random graphs is [MS05, MS06b, MS06cJ where the divergence of the point- 
to-set correlation length is linked with divergence of the equilibration time of the Glauber 
dynamics. This suggests that beyond the clustering transition the Monte Carlo sampling 
(or maybe even sampling in general) will be a hard task. 

Note also that in the mathematical literature the Glauber dynamics is often studied. 
Many results exist about the so-called rapid mixing of the associated Markov chain [Sin93j . 
But the rapid mixing questions equilibration in polynomial time, whereas in physics the 
relevant time scale is linear. Moreover rapid mixing is defined as convergence to the 
equilibrium measure from any possible initial conditions, whereas in physics of glasses 
the notion of a typical initial condition should be used instead. 



2.1.3 Compendium of the 1RSB cavity equations 

We review the 1RSB equations on a general CSP. The order parameter is a probability 
distribution of the cavity field (BP message) tp a ^ 1 = (^q~^ 1 , . . . , ipg^i). The self-consistent 
equation for p a ^ % reads 

j£da—i badj—a 

(2.24) 

where the function •*}) and the term Z^ % are defined by the BP equation (11.17I) . 

Zi~* 1 is a normalization constant. 

The associated thermodynamical potential (12.151) is computed as 

a i 

Z a+da fjjfj [dP b ~' i (^ l )](Z a+da ) m , (2.25b) 
i£c9ct badi—a 

Z { = J Yl [dP^ty*-*)] (Z 1 )™ , (2.25c) 

where the terms Z a+da and Z % are the partition sum contributions defined in (11.191) . 

The logarithm of the number of states divided by the system size defines the com- 
plexity function S. Inversely the number of states is e NVl . At finite temperature the 
complexity of states with a given internal free energy is a Legendre transformation of the 
potential $(m) 

$(m) = -Pmf + £(/) , (2.26) 
Useful relations between the free energy, complexity and potential $ are 

d f E(f) = (3m, d m $(m) = -Pf, m 2 d m ^- = -Y,. (2.27) 



ni 
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At zero energy, E — 0, and zero temperature, j3 —>■ oo, the free energy becomes entropy 
—/3f — > s. Then the complexity is a function of the internal entropy of states and ( 12. 26ft 
becomes 

$(m) =ms + S(s), (2.28) 

with 

$f 777,") 

<9 s £(s) = -m , <9 m $(m) = s , m 2 <9 m ^^ = -£ . (2.29) 

m 

This is called the entropic zero temperature limit. The internal entropy is expressed as 

s = a7 ( E A ^ a+9a - X> " ^ A5 ' ( 2 ' 3 °) 

a 2 

where A5* a+9a (AS* 1 resp.) is an internal entropy shift when the constraint a and all its 
neighbour (the variable i resp.) are added to the graph. 

Aqa+da _ IJheaJhe^a [dP^(^)] (Z°+°*) m log Z°+*> 

- ;n ae ajdp-(^)](^) m ' [ } 

In the energetic zero temperature limit, described in sec. 11.61 for zero energy, the Parisi 
parameter y = (3m is kept constant, thus m —>■ 0. The free energy then converges to the 
energy, and (12.261) becomes 

$(y) = -2/e + £(e), (2.32) 

where the complexity is this time a function of the energy density e. The survey propa- 
gation equations generalized to nonzero y are called the SP-y equations. 

Equations (12. 24112. 3T|) are defined on a single instance of the constraint satisfaction 
problem. Averages V over the graph ensemble are obtained in a similar manner as in 
sec. 11.5.31 for the replica symmetric solution. 

= E [n&eo] / II n {dp[^(^)]}4 p w -mp ji m})] , 

{k} h J i=l ii=l 

(2.33) 

where in the sum over {Zj}, i e {1, . . . , — 1}, and the functional JF 2 is defined by 
(I2.24p . Analogical expression holds for the average of the complexity or internal entropy. 
A general method to solve the equation (12.331) is the population of populations described 
in appendix IE. 51 



2.2 Geometrical definitions of clusters 

Up to now we were describing clusters, i.e., partitions of the space of solutions, in a 
very abstract way which was defined only in the thermodynamical limit. We showed 
how to compute the number of clusters of a given size (internal entropy) (I2.28p . and we 
argued that the description makes sense if the point-to-set correlation (12.221) decays to 
zero within almost every cluster of that size. In this last sense clusters are what we would 
call in statistical physics pure equilibrium states. 
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On a very intuitive level, cluster are groups of nearby solutions which are in some sense 
separated from each other. Several geometrical definitions are used in the literature, we 
want to review the most common ones and state their relation to the definition above. We 
want to stress that it is not know whether any of the geometric definitions is equivalent 
to the description given above and used usually in the statistical physics literature. 

Strong geometrical separation, ^-satisfiability — First rigorous proofs of exis- 
tence of an exponential number of clusters of solutions in the random K-SAT were based 
on the concept of x-satisfiability. Two solutions are at distance x if they differ in exactly 
xN variables. A formula is said x-satisfiable if there is a pair of solutions at distance x, 
and x-unsatisfiable if there is not. 

Mora, Mezard and Zecchina [MMZ051 IDMMZ08] managed to prove that for K > 8 and 
a constraint density a near enough to the satisfiability threshold the formulas are almost 
surely x-satisfiable for x < Xq, almost surely x-unsatisfiable for x\ < x < x 2 , and almost 
surely x-satisfiable at x 3 < x < x 4 , where obviously < x < X\ < x 2 < x 3 < x 4 < 1. 
This means that at least two well separated clusters of solutions exist. Proving that 
there is an exponentially smaller number of pairs of solutions at distances x < x\ than at 
distances x > x 2 leads to the conclusion that an exponential number of well geometrically 
separated clusters exists [ART06J. 

However, the x-satisfiability gives too strong conditions of separability. This is illus- 
trated for example in the XOR-SAT problem [MM06bj . It is still an open question if 
there is or not a gap in the x-satisfiability in the random 3-SAT near to the satisfiability 
threshold. 

Connected-components clusters — Another popular choice of a geometrical defini- 
tion of clusters is that clusters are connected components in a graph where every solution 
is a vertex and solutions which differ in d or less variables are connected. The distance 
d is often said to be any sub-extensive (in the number of variables N) distance, that is 
d = o(N). However, such a rule is not very practical for numerical investigations. 

In iT-SAT, in fact, d — 1 seems to be a more reasonable choice. There are two reasons: 
First, clusters defined via d — 1 have correct "whitening" properties as we explain in the 
next paragraph. Second, we numerically investigated the complexity of d = 1 connected- 
components clusters, fig. 12.21 right, and the agreement with the total number of clusters 
computed from (12.281) at in = is strikingly good. In particular, near to the satisfiability 
threshold a > 4.15, where the 1RSB result for the total complexity function is believed 
to be correct (stable) |MPRT04| . 

Formally, connected-components clusters have no reason to be equivalent to the notion 
of pure states. They are not able to reproduce purely entropic separation between clusters, 
which might exist in models like 3-SAT. However, fig. 12.21 suggests that there is more in 
this definition than it might seem at a first glance. 

Whitening-core clusters — We define the whitening of a solution as iterations of 
the warning propagation equations ( 11.351) initialized in the solution. The fixed point is 
then called the whitening core. Note, that the whitening core is well defined in the sense 
that the fixed point of the warning propagation initialized in a solution does not depend 
on the order in which the messages were updated. A whitening core is called trivial if all 
the warning messages are 0, that is "I do not care". 
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The 1RSB equations at m — 0, which give the total complexity function, can be 
derived as belief propagation counting of all possible whitening cores |MZ02l [BZ04, 
MMW07J. Thus another reasonable definition of clusters is that two solutions belong 
to the same cluster if and only if their whitening core is identical. In fig. 12.21 left we 
plot numerically computed complexity of the whitening-core clusters compared to the 
complexity computed from ( 12. 28ft at m = 0. The agreement is again good, in particular 
near to the satisfiability threshold, a > 4.15, where the SP gives a correct result. 

The d — 1 connected-components clusters share the property that all the solution 
from one clusters have the same whitening core. Proof: If this would not be true then 
there have to exist a pair of solutions which do not have the same whitening core but 
differ in only one variable, this is not possible because then the whitening could be started 
in that variable. 

Note, however, that the definition of whitening-core clusters put all the solutions with 
a trivial whitening core into one cluster. This is not correct as, at least near to the 
clustering threshold, there are many pure states with a trivial whitening core. This is 
closely connected to the properties of frozen variables which will be discussed in chapter HI 




density of constraints density of constraints 



Figure 2.2: Right: Complexity of the connected-components clusters. Left: Complexity 
of the whitening-core clusters. Both compared to the complexity computed from the 
survey propagation equations. The data for the SP complexity are courtesy of Stephan 
Mertens, from |MMZ06j . 



Enumeration of clusters in 3-SAT: the numerical method — In order to obtain 
the data in fig. 12.21 we generate instances of the random 3-SAT problem with iV variables 
and M clauses, constraint density is then a = M/N. We count number of solutions 
in A = 999 random instances and choose the median one where we count the number 
of connected-components and whitening-core clusters S. This is repeated B = 1000 
times. The average complexity is then computed as S = J2f=i ^°S^i/(BN), if the median 
instance was unsatisfiable then we count zero to the average, that is if all the B instances 
are unsatisfiable then the complexity is zero. We do such a non-traditional sampling to 
avoid rare instances with very many solutions, which we would not be able to cluster. 
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2.3 Physical properties of the clustered phase 

Let us give a summary of the properties of the clustered phase, also called the dynamical 
1RSB phase. We describe only the situation when E(m = 1) > (I2.28p . when the 
opposite is true the properties are completely different as we will discuss in the next 
chapter [31 

The complexity function computed from (12.281) is the log-number of clusters of a given 
internal entropy. If a solution is chosen uniformly at random it will almost surely belong 
to a cluster with entropy s* such that E(s) + s is maximized in s*, d s Il(s*) = —1, that is 
m = 1. At m = 1 the total entropy + s* = $(m = 1). The replicated free entropy 

at $(m = 1) is equal to the replica symmetric entropy. Thus the total entropy in the 
dynamical 1RSB phase is equal to the RS entropy. Also the marginal probabilities at 
m = 1 are equal to the replica symmetric ones 

J dP^(^) ^ = (Vrs^ if m=l. (2.34) 

Thus the clustering transition is not a phase transition in the Ehrenfest sense, because 
the thermodynamical potential, entropy in our case, in analytical at the transition. 

The overlap (or here distance) distribution, which is often used to describe the spin 
glass phase, is also trivial and equal to the replica symmetric one in the dynamical 1RSB 
phase. Indeed, if exponentially many clusters are needed to cover almost all solutions, 
then the probability that two solutions happen to belong to the same cluster is zero. 

The correlation function between two variables at a distance (shortest path in the 
graph) d is defined as (siSj) c = ||/i(s», Sj) — //(si)//(sj)||xv- The variance of the overlap 
distribution, which is negligible compared to 1 as we explained, can be expressed as 
J2ij( s i s j)c/N 2 , and thus the two-point correlation have to decay faster with distance 
than the number on vertices at that distance is growing. This means in particular that 
two neighbours of a node % are independent if we condition on the value of i, this is 
again consistent with the fact that the belief propagation equations predict correct total 
entropy and marginal probabilities. 

So far nothing is different form the replica symmetric phase. It is thus not straightfor- 
ward to recognize the dynamical 1RSB phase based on the original replica computation. 
Presence of this phase was discovered and discussed in |KT87al IKT87bj . Later purely 
static methods were developed to identify this phase. The most remarkable is perhaps 
the e-coupling and the "potential" of [FP951 IFF57j . 

In our setting the main difference between the replica symmetric phase and the dy- 
namical 1RSB phase is that in the later the point-to-set correlations do not decay to 
zero. Consequently the equilibration time of the local Monte Carlo dynamics diverges 
and Monte Carlo sampling becomes difficult [MS06bJ. 

2.4 Is the clustered phase algorithmically hard? 

Clustering has important implications for the dynamical behaviour. It slows down the 
equilibration and thus uniform sampling of solutions via local single spin flip Monte Carlo 
is not possible, or exponentially slow, beyond the dynamical threshold. But finding one 
solution is a much simpler problem than sampling. 
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Analytic arguments — In the 3-coloring of Erdos-Renyi graphs the clustering thresh- 
old is Cd = 4, as at this point the spin glass susceptibility diverges, see appendix O In 
the terms of the reconstruction problem the Kesten-Stigum |KS66al KS(i(ih bound is 
sharp. On the other hand Achlioptas and Moore [AM03J proved that a simple heuristic 
algorithm is able to find a solution in average polynomial time up to at least c = 4.03. 
This shows that the RSB phase is not necessarily hard. 

A similar observation was made in the l-in-3 SAT problem in [ZDEB-3]. There is a 
region in the values of the average density of constraints and the probability of negating 
a variable in a clause in which the replica symmetric solution is unstable and yet the 
unit clause propagation algorithm with the short clause heuristics was proven to find a 
solution in polynomial average time. 

We should mention a common contra- argument; which is that in the above mentioned 
regions the 1RSB approach might not be correct, and the presumably full-RSB phase 
|Par80cj is more "transparent" for the dynamics of algorithms, see e.g. [MRT04J. However, 
at least in the 3-coloring, the 1RSB approach seems to be correct in the interval in 
question, as we argue in appendix [D] 



Stochastic local search — There is a lot of numerical evidence that relatively simple 
single spin flip stochastic local search algorithms are able to find solutions in linear time 
deep in the clustered region. Examples of works where performance of such algorithms 
was analyzed are |KK07} ISAO051 IAA06} IAAA+07] . In fig. EH] we give an example of 
performance of the ASAT algorithm |AA06j in 4-coloring of Erdos-Renyi random graphs 
[ZDEB-5]. The algorithm is described in appendix IF. 2. 2[ In the 4-coloring ASAT is able 
to find solutions in linear time beyond the clustering transition cj = 8.35 
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Figure 2.3: The performance of the ASAT algorithm in the 4-coloring of random Erdos- 
Renyi graphs. Left: The energy density plotted against the number of steps per variable. 
Right: The average running time (per variable) as a function of the connectivity. The 
time does not diverge at the clustering transition q, but beyond it. The other phase 
transitions marked are the condensation transition c c (chap. [3]) the rigidity transition c r 
(chap. HJ) and the colorability threshold c s 



Simulated annealing — There is no paradox in the observations above. Quantitative 
statements are, however, difficult to make. Let us describe on an intuitive level the 
behaviour of an algorithm (dynamics) which satisfies the detailed balance condition and 
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thus in infinite time samples uniformly from the uniform measure ( 12.11) . We think for 
example about the simulated annealing [KGV83J. Above the dynamical temperature 
T d corresponding to an energy E d the point-to-set correlation function f 1 2 . 2 2 j) decay fast 
and thus simulated annealing is able to reach the equilibrium. Below temperature T d 
this is not the case anymore and the dynamics is stuck for a very long time in one 
of the clusters, states. But the bottom of this state -^bottom lies lower than Ed, thus 
when lowering the temperature the average energy seen by the simulated annealing also 
decreases. If -Ebottom — then the algorithm will find a solution. It is not known how 
to compute ^bottom in general. Sometimes, far from the clustering transition, the iso- 
complexity approach [MRT04j gives a lower bound on bottom- But in general, as far as 
we know, there is no argument saying bottom > 0. This picture can be substantiated for 
several simple models as the spherical p-spin model [CK93J or the random subcubes model 
[ZDEB-8]. The connection with the optimization problems was remarked in [KK07] . 

For the stochastic local search algorithm, which does not satisfy the detailed balanced 
condition, the situation might be similar. At a point the algorithm is stuck in a cluster, 
but if this cluster goes down to the zero energy then it might be able to find solutions 
even in the clustered phase. 

However, the current understanding of the dynamics of the mean field glassy systems 
is far from complete. More studies are needed to understand better the link between the 
static clustered phase and the dynamical behaviour. 
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Chapter 3 
Condensation 



In this chapter we will describe the so-called condensed clustered phase. Before turning 
to the models of our interest we present the random subcubes model [ZDEB-8], where the 
condensation of clusters can be understood on a very elementary probabilistic level. After 
mentioning that the condensed phase is in fact very well known in spin glasses we describe 
the Poisson-Dirichlet process which determines the distribution of sizes of clusters in that 
phase. Further, we discuss general properties of the condensed phase in random CSPs. 
And finally we address our original question and conclude that the condensation is not 
much significant for the hardness of finding a solution [ZDEB-5]. 



3.1 Condensation in a toy model of random subcubes 

The random-sub cubes model [ZDEB-8] is defined by its solution space S C {0, 1}^; we 
define S as the union of [2( 1_a ) JV "J random clusters (where |_^J denotes the integer value 
of x). A random cluster A being defined as: 

A = {a\Vie{l,...,N},o- t enf}, (3.1) 

where 7i A is a random mapping: 

n A :{l,...,N} — > {{0},{1},{0,1}}, (3.2) 
i nf, (3.3) 

such that for each variable i, nf = {0} with probability p/2, {1} with probability p/2, 
and {0, 1} with probability 1 — p. A cluster is here a random subcube of {0, 1}^. If 
nf = {0} or {1}, variable % is said "frozen" in A; otherwise it is said "free" in A. In this 
model one given configuration a might belong to zero, one or several clusters. 

We describe the static properties of the set of solutions S in the random-subcubes 
model in the thermodynamic limit N — > oo (the two parameters < a < 1 and < p < 1 
being fixed and independent of N). The internal entropy s of a cluster A is defined as 
jj log 2 \ A\, i.e., the fraction of free variables in A. The probability V(s) that a cluster has 
internal entropy s follows the binomial distribution 

V(s)=(")(l-pyV- )N - (3-4) 



Then the number of clusters of entropy s, denoted Af(s), is with high probability 

1. .r/ \ f E(s) = 1 -a-D(s || l-p) if£(s)>0, lo r , 

hm — \og 2 Af(s) = < w v 11 + i. ~ (3-5) 

tv^oo N I -oo otherwise, v ' 
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where D(x \\ y) = x log 2 ^ + (1 — x) log 2 \^ is the binary Kullback-Leibler divergence. 

We compute the total entropy s to t = jr log 2 \ S\. First note that a random configura- 
tion belongs on average to 2 Af ( 1 ~ a )(l - 'fj N clusters. Therefore, if 

a < a d = log 2 (2 -p), (3.6) 

then with high probability the total entropy is s to t = 1- 

Now assume a > ad- The total entropy is given by a saddle-point estimation: 

J2 2 S(A)JV = [1 + o(l)]N [ ds 2 N ^ +S \ (3.7) 
whence s to t = max [S(s) + s | S(s) > 0]. (3.8) 

s 

We denote by s* = argmaxjE(s) + s \ E(s) > 0] the fraction of free variables in the 
clusters that dominate the sum. Note that our estimation is valid (there is no double 
counting) since in every cluster the fraction of solutions belonging to more than one 
cluster is exponentially small as long as a > 

Define s = 2(1 —p)/(2—p) such that <9 s S(s) = —1. The complexity of clusters with 
entropy s reads: 

S(5) = -^-+log 2 (2-p)-a. (3.9) 
2 — p 

s maximizes eq. (13.81) as long as S(s) > 0, that is if 

a<a c = + log 2 (2-p). (3.10) 
{2-p) 

Then the total entropy reads 

s tot = 1 - ol + log 2 (2 - p) for a < a c . (3.11) 



For a > a c , the maximum in (13.81) is realized by the largest possible cluster entropy s max , 
which is given by the largest root of S(s). Then s to t = s* = s max . We will show in the 
next section that in such a case almost all solutions belong to only a finite number of 
largest clusters. This phase is thus called condensed, in the sense that almost all solutions 
are "condensed" in a small number of clusters. 

In summary, for a fixed value of the parameter p, and for increasing values of a, four 
different phases can be distinguished: 

(a) Liquid (replica symmetric) phase, a < ad- almost all configurations are solutions. 

(b) Clustered (dynamical 1RSB) phase with many states, ad < a < a c : an exponential 
number of clusters is needed to cover almost all the solutions. 

(c) Condensed clustered phase, a c < a < 1: a finite number of the biggest clusters 
covers almost all the solutions. 



(d) Unsatisfiable phase, a > 1: no cluster, hence no solution, exists. 



3.2. NEW IN CSPS, WELL KNOWN IN SPIN GLASSES 
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Figure 3.1: The complexity function in the random subcubes model, E(s) (13.51) . for 
p = 0.8 and several values of a. The red dots mark the dominating clusters s*,S(s*). 
For p = 0.8 the dynamical transition aa ~ 0.263 is far away from the plotted values, the 
condensation transition is a c ~ 0.930, the satisfiability a s = 1. 



3.2 New in CSPs, well known in spin glasses 

The complexity function E(s) (12. 26ft in random CSPs is counting the logarithm of the 
number of clusters per variable which have internal entropy s per variable. We define 
dominating clusters in the same way as in the random subcubes model, that is clusters 
of entropy s* such that 

s* = arg max [E(s) + si . (3.12) 

s,E(s)>0 

In chap. |2]we discussed properties of the dynamical 1RSB phase, that is when £(s*) > 0, 
in other words when there are exponentially many dominating clusters. 

The condensed phase with S(s*) = 0, described in the random subcubes model, exists 
also in random CSPs. And in the context of constraint satisfaction problems it was first 
computed and discussed in |MPR05j and [ZDEB-4]. However, historically it was the 
condensed phase where the 1RSB solution was first worked out |Par8 0c|. A very simple 
example of condensation can also be found in the random energy model [Der801 IDer81j . 
As we discussed in the previous chapter [2J the dynamical 1RSB phase is well hidden 
within the replica solution — the total entropy is equal to the replica symmetric entropy, 
the overlap distribution is trivial and the two-point correlation functions decay to zero 
etc. All this changes in the condensed phase. 

A small digression to the physics of glasses: In structural glasses, the analog of the 
condensation transition is well known for a long time, its discovery goes back to Kauz- 
mann in 1948 who studied the configurational entropy of glassy materials. Configurational 
entropy is the difference between the total (experimentally measured) entropy and the 
entropy of a solid material, this thus corresponds to the complexity function. In the so 
called fragile structural glasses |Ang95| the extrapolated configurational entropy becomes 
zero at a positive temperature, nowadays called the Kauzmann temperature. The Kauz- 
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mann temperature in the real glasses is, however, only extrapolation. The equilibration 
time in glasses exceeds the observation time high above the Kauzmann temperature. It 
is a widely discussed question if there exists a true phase transition at the Kauzmann 
temperature or not, for a recent discussion see fDSOlJ. 



Why does Parisi maximize the replicated free energy? As we said, it is the con- 
densed phase which was originally described by Parisi and his one-step replica symmetry 
breaking solution |Par80c] . Let us now briefly clarify the relation to the replica solution, 
similar reasoning first appeared in |Mon95j . In sec. 12.11 we called the Legendre trans- 
form of the complexity function the replicated free entropy $(m) (I2.26p . In the replica 
approach the replicated entropy Q(m) = $(m)/m is computed. From (I2.28P follows 



0( m ) = s + —-^ where J- ; = V • ( 3 - 13 ) 

m dm m 2 



Thus, in the condensed phase, computing the largest root of the function in order 

to maximize the total entropy, is equivalent to extremizing the replicated entropy Q(m). 
Moreover, as the function is concave and the parameter m is minus its slope this 
extrema have to be a minima. Thus in the Parisi's replica solution we have to minimize the 
replicated entropy function with respect to the parameter m. If a temperature is involved 
then this becomes a maximization of the replicated free energy, this might have seem 
contra-intuitive in the original solution, but it comes out very naturally in our approach. 
Other physical interpretation of the maximization was proposed e.g. in [Jan05j. 



3.3 Relative sizes of clusters in the condensed phase 

What is the number of dominating clusters in the condensed phase and what are their 
relative sizes? So far we know that the entropy per variable of the dominating states is 
s* + o(l) and that their number is sub-exponential, S(s*) = 0. But much more can be 
said based on purely probabilistic considerations. 

Consider that the total number of clusters M is exponentially large in the system size 
N, and that N —>■ oo. Let the log-number of clusters of a given entropy be distributed 
according to an analytic function E(s). Denote — m* = <9 s £(s*), in the condensed phase 
< m* < 1. Denote the size of the a th largest cluster e Ars * +Aa , A Q = 0(1). The 
probability that there is a cluster of size between e Ns * + and e Ns *+ A + dA ^ /\ ^> dA, j s 
e _m * A dA, in other words points A a are constructed from a Poissonian process with rate 
e -m*A [I] R e i a tj ve s i ze f the a th largest cluster is defined as 

^ = ^ A • ( 3 - 14 ) 
E 7 =i eA7 

Point process w a which is constructed as described above is in mathematics called the 
Poisson-Dirichlet process [PY97] . The connection between this process and the relative 



1 Note that in the random subcubes model the numbers (Ns* + A Q )log(2) are integers equal to the 
number of free variables in the cluster A a . Then A a are discrete and some of the properties of the 
resulting process might be different from the Poisson-Dirichlet. 
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weights of states in the mean field models of spin glasses was (on a non-rigorous level) 
understood in [MPV85j . for more mathematical review see 

Any moment of any w a can be computed from the generating function |PY97j 



E[exp(-A/0] = e- A m *(A) a -V m *(A)- Q 
where A > and the functions m * and ip m * are defined as 

0m* (A) 



m 



e~ Xx x~^~ m * dx . 



1 + m* (1 -e 
'o 



- Xx )x- l - m * dx . 



(3.15) 

(3.16a) 
(3.16b) 



The second moments can be used to express the average probability Y that two random 
solutions belong to the same cluster 



a=l 



1 — m 



(3.17) 



This was originally derived in [MPV85J 

Another useful relation |PY97j is that the ratio of two consequent points R a = 
w a+ i/w a , a = 1,2, ...,J\f is distributed as am*^™*" 1 . In particular its expectation 
is E[i2 a ] = am* /(I + am*) and the random variables R a are mutually independent. We 
used these relation to obtain data in figure 13.21 




m" 



Figure 3.2: The fractions of solutions covered by the largest clusters as a function of 
parameter m*. The lower curve is related to the size of the largest clusters as 1/E[1/Wl] = 
1 — m*. The following curves are related to the size of i largest clusters, their distances 
are E[R a ]E[R a ^] . . .E[#i](l - m*). 



2 To avoid confusion, note that the Poisson-Dirichlet process we are interested in is the PD(m*,0) 
in the notation of [PY97] . In the mathematical literature, it is often referred to the PD(0,#) without 
indexing by the two parameters. 
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From the properties of the Poisson-Dirichlet process, it follows that an arbitrary large 
fraction of the solutions can be covered by a finite number of clusters. When m* is near to 
zero, that is near to the satisfiability threshold, the largest cluster covers a large fraction 
of solutions. On the other side, when m* is near to one, that is near to the condensation 
transition, very many (but finite in N) clusters are needed to cover a given fraction of 
solutions. 



3.4 Condensed phase in random CSPs 

The total entropy in the condensed phase is strictly smaller than the replica symmetric 
entropy, s to t = s* < srs- At the condensation transition c c the total entropy is non- 
analytic, it has a discontinuity in the second derivative. This can be seen easily for 
example from the expressions for the random subcubes model. At a finite temperature 
the discontinuity in the second derivative of the free energy corresponds to a jump in 
the specific heat. The parameter m* — 1 at the condensation transition and decreases 
monotonously to m* = at the satisfiability threshold. 



Concept of self-averaging — In the physics of disordered systems the self-averaging 
is a crucial concept. We say that quantity A measured on a system (graph) of N variables 
is self-averaging if in the limit iV — > oo 

[E(^j] >°' (3 - 18) 

where the average E [ • ] is over all the disorder in the system. In other words a quantity 
is self- averaging if its value on a typical large system is equal to the average value. By 
computing the average value we thus describe faithfully the typical large system. And 
also measuring A on a single large system is enough to represent the whole ensemble. On 
finite-dimensional lattices and off criticality extensive quantities are always self-averaging. 
This can be shown by building the large lattice from smaller blocks, the additivity of an 
extensive quantity and the central limit theorem then ensures the self-averaging. At the 
critical point, on a mean field lattice (fully connected or tree-like) or for non-extensive 
quantities the answer whether A is self-averaging or not becomes nontrivial. 

In the condensed phase quantities which involve the weights of clusters are not self- 
averaging. This arises from the fact that the dominating clusters are different in every 
realization of the system. Statistical properties of many quantities of interest can be 
described from the Poisson-Dirichlet process. 



Overlap distribution — The overlap between two solutions is defined as one minus 
the Hamming distance 

?(w,w) = ^E^yi). (3.i9) 

i=i 

The overlap between two solutions belonging to two different dominating clusters is qo, 
and between two solutions belonging to the same dominating cluster q 1 . Values q and qi 
are self-averaging. The distribution of overlaps in the limit N — > oo can thus be written 

as 

P(q) =w5(q- Ql ) + (l-w) 5(q - q ) , (3.20) 
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where the weight w is the probability that two random solutions belong to the same 
cluster. Thus w = Yla=i w "a-> wnere w a are weights of the clusters (13. 1411 given by the 
Poisson-Dirichlet process. The weights change from realization to realization, w is thus 
not a self-averaging quantity, its typical value fluctuates around the mean K(w) = 1 — m* 
computed in (I3.17P . The distribution of the random variable w is also known |MPS+84j . 

Two-point correlation functions — The variance of the overlap distribution is 

varg= / q 2 P(q)dq— [ qP(q)dq = w(l — w)(qi — go) 2 • (3-21) 



At the same time the variance is equal to 

var 9 = -pp Y Y s ^ ~ M«i)Msj) I ~ Y Y M Si ' s ) - fi(si)fi(s ) | , (3.22) 

i,j Si,Sj i s 4 , so 

where so is a typical variable in the random graph. If we consider that the two-point 
correlation function is of order one up to a correlation length £ and zero after that we get 

varg^^c ? , (3.23) 

where c is approximately the branching factor. In the condensed phase the variance of 
the overlap is of order one thus the correlation length has to be of order logiV. But the 
shortest path between two random variables is also of order logiV thus the two-point 
correlations cannot be neglected in the condensed phase. 

If two-point correlations cannot be neglected then the derivation of belief propagation 
equations fll.16am.16bj) is not valid, because we supposed that the neighbours of a node 



i are independent when we condition on the value of i. It is thus not surprising that the 
value to which the BP equations converge (if they do), does not correspond to the true 
marginal probability. Formally, the BP fixed point corresponds to the 1RSB equations 
at m = 1, but in the condensed phase m* < 1. 

In fact, the probability distribution of the true marginal probabilities is another ex- 
ample of a non self-averaging quantity. It again depends on the realization of the Poisson- 
Dirichlet process. 



3.5 Is the condensed phase algorithmically hard? 

From the algorithmic point of view the only important difference between the dynamical 
1RSB phase and the condensed phase is that in the condensed phase the belief propaga- 
tion does not estimate correctly the asymptotic marginal probabilities. In the condensed 
phase, the total entropy cannot be estimated from the BP equations either, thus ap- 
proximative counting and sampling of solutions will probably be even harder than in the 
dynamical 1RSB phase. 

Concerning the hardness of finding a solution we might expect that the incorrectness 
of the belief propagation estimates of marginals will play a certain role. However, we 
used the belief propagation maximal decimation as described in appendix IF. 1.21 in the 
3- and 4-coloring, see fig. 13.31 And this algorithm does not seem to have any problem 
to pass the condensation transition in both these cases. In particular, in the 3-coloring 
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Figure 3.3: The performance of the maximal BP decimation algorithm, described in 
appendix IF. 1 .2^ in the 3-coloring (left) and the 4-coloring (right) of random graphs. 
This algorithm is able to color random graphs beyond both the clustering Cd and the 
condensation c c transitions in 3- and 4-coloring. 



the gap between the condensation threshold c c = 4 and the limit of performance of the 
BP decimation c ~ 4.55 is huge. The rigidity transition c r , defined in chapter HI and the 
colorability threshold c s are also marked for comparison in fig. 13.31 

The condensation transition thus does not seem to play any significant role for the 
computational hardness of finding a solution. 



Chapter 4 



Freezing 

The previous two chapters describe recent contributions to the understanding of the clus- 
tering and condensation of solutions in random constraint satisfaction problems. Both 
these concepts are well known and widely discussed in the mean field theory of glasses 
and spin glasses for at least a quarter of a century. 

The concept of freezing of variables appeared in the studies of optimization problems, 
that is systems at zero temperature (or infinite pressure). In this chapter we first define 
the freezing of variables, clusters and solutions, and discuss its properties both in the 
thermodynamical limit and on finite-size instances. Then we explain how to describe the 
frozen variables within the one-step replica symmetry breaking approach and we define 
several possible phase transition associated to the freezing. To simplify the picture we de- 
fine and solve the "completely frozen" locked constraint satisfaction problem where every 
cluster contains only one configuration. Finally we give several arguments about connec- 
tion between the freezing and the average computational hardness. Results of this section 
are mostly original and were published in [ZDEB-5, ZDEB-7, ZDEB-10, ZDEB-9]. 

4.1 Frozen variables 

Consider a set of solutions S of a given instance of a constraint satisfaction problem. 
Define that a variable i is frozen in the set of solutions A C S if it is assigned the same 
value in all the solutions in the set. If an extensive number of variables is frozen in the 
set A, then we call A and all the solutions in A frozen, otherwise A and all the solutions 
in A are called soft (unfrozen). 

A first observation is that the set of all solutions S is not frozen in the satisfiable 
phase. If it would be then adding one constraint, i.e., increasing the constraint density 
by 1/N, would make the formula unsatisfiable with a finite probability, that would be in 
a contradiction with the sharpness of the satisfiability threshold. The backbone is made 
of variables frozen in the set of ground states. An extensive backbone can thus exist only 
in the unsatisfiable phase. Already in |MZK + 99b] it was argued that there might be a 
connection between the backbone and the computational hardness of the problem. The 
sueg estion of [MZK+99b] was that if the fraction of variables covered by the backbone is 
discontinuous at the satisfiability transition then it is hard to find satisfying assignments 
on highly constrained but still satisfiable instances. On the other hand if the backbone 
appears continuously the problem is easy in the satisfiable phase. This was based on the 
replica symmetric solution of the random i^-SAT which does not describe fully the phase 
space, in spite of that the relation between the existence of frozen variables inside clusters 
and the algorithmical hardness seems to be deep and we will develop it in this chapter. 
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4.1.1 Whitening: A way to tell if solutions are frozen 

How to recognize if clusters have frozen variables or not. Or how to recognize if a given 
solution belongs to a frozen cluster or not. An iterative procedure called whitening 
|Par02a| gives an answer to these questions. 

Given a formula of a CSP and one of its solutions {si} G { — 1, 1}^, i — 1, • • • , N, the 
whitening of the solution is defined as iterations of the warning propagation equations 
( 11.35)) initialized on the solution. That is, for a binary CSP = Sj, and u^ t l is 

computed according to eq. (|1.35bj) . Note that the fixed point of the whitening does not 



depend on the order in which the warnings are updated. Indeed, during the iterations 
the only changes in warnings are from non-zero values to zero values. The fixed point is 
called the whitening core of the solution. The whitening core is called trivial if all the 
warnings are equal to 0, and nontrivial otherwise. 

In the K-SAT problem whitening can be reformulated in a very natural way: Start 
with the solution {sj}, assign iteratively a "*" (joker) to variables which belong only to 
clauses which are already satisfied by another variable or already contain a * variable. 
On a general CSP such procedure is not equivalent to the whitening, and the warning 
propagation definition has to be used instead in order to obtain all the desired properties 
and relations to the 1RSB solution. 

We now argue that if the 1RSB solution is correct, then frozen variables in the cluster, 
to which solution {sj} belongs, asymptotically correspond to variables for which in the 
whitening core the total warning h l ^ (jl.37jl . Thus whitening can be used to decide 
if the solution {sj} belongs to a frozen cluster without knowing all the solutions in that 
cluster. The first step to show this property is, as in sec. 12.1.11 to consider the CSP 
on a tree with given boundary conditions which are compatible with a non-empty set of 
solutions S in the interior of the tree. Starting on the leaves we compute iteratively the 
warnings (11.351) down to the root. Variables which have at least one non-zero incoming 
warning are frozen in the set S. The correctness of the 1RSB approach on a tree-like 
graph means that the picture on a tree captures properly all the asymptotic properties. 
In particular, the whitening core determines the set of frozen variables on typical large 
instances of the problem. The correctness of the 1RSB solution is an essential assumption 
for the above statement. Because all the long-range correlations decay within one cluster 
the warnings in the whitening core are independent in the absence of %. Thus there 
truly exist solutions in that cluster in which the variable i takes all the values allowed by 
the warnings. And on the other hand, if a value is not allowed by the warnings there is no 
solution where i would be taking this value. For consistency, all solutions in one cluster 
have to have the same whitening core. However, two different clusters can have the same 
whitening core. The most important example are all the soft (not frozen) clusters that 
all have the trivial whitening core. 

Whitening, as the iterative fixed point of the warning propagation, may be defined 
not only for a solution but for any configuration. In this way one may find blocking 
metastable states. For some preliminary numerical considerations see [SAO05J. 



4.1.2 Freezing on finite size instances 

The definition of whitening is applicable to any (non-random, small, etc.) instance. 
What does then remain from the asymptotic correspondence between frozen variables 
and whitening cores? 
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Consider now clusters as connected components in the graph where all solutions are 
nodes and where edges are between solutions which differ in only one variable, as in 
sec. 12.21 Several questions arise about this definition: 

• Do all the solutions in the connected-components cluster have the same whitening 
core? The answer is yes. If there were two solutions with different whitening cores 
which can be connected by a chain of single- variable flips, then along this chain there 
would exist a pair of solutions which differ in only one variable i and have different 
whitening cores. But this is not possible, as the fixed point of the whitening does 
not depend on the order in which the warnings were updated, and one could thus 
start the whitening by setting warnings h l ^ a = 0. 

• Does the whitening core of a connected-components cluster correspond to the set 
of frozen variables? The answer is: If in the whitening core h l ^ (11.371) then the 
variable % is frozen in the connected-components cluster. Proof: If such a variable 
i is not frozen, then there have to exist a pair of solutions which differ only in the 
value of this variable. Then all the constraints around % have to be compatible 
with both these values, this would be in contradiction with ft 1 ^ 0. On the other 
hand, if in the whitening core h l = then the variable % might still be frozen in the 
connected-components cluster on a general instance, because correlations which are 
not considered by the 1RSB solution may play a role. 

Consider now clusters as the set of all solutions which share the same whitening 
core. Whitening-core clusters are aggregations of the connected-components clusters. In 
particular, all the solutions with a trivial whitening core, which might correspond to 
exponentially many pure states, are put together. 

• What is the set of frozen variables in the whitening-core clusters? The answer is: 
Again if in the whitening core h % ^ then the variable i is frozen in the whitening- 
core cluster. In principle, one whitening-core cluster could be an union of several 
connected-components cluster, but i is frozen to the same value in each of them. 
The inverse is not correct in general. On finite size instances some variables with a 
zero warning ft = might be frozen in the whitening-core cluster. 

• Can there be a fixed point of the warning propagation (11. 35ft corresponding to zero 
energy (I1.38P which is not compatible with any solution? The answer is yes. And 
such fixed points were observed in [BZ041 IMMW071 IKSS07bj . Again if the 1RSB 
solution is correct then in the thermodynamical limit these "fake" fixed points are 
negligible. 

4.1.3 Freezing transition in 3-SAT - exhaustive enumeration 

Before turning to the cavity description of frozen clusters we investigate the freezing 
transition in the random 3-SAT numerically. We define the freezing transition, a/, as 
the smallest density of constraints a such that the whitening core of all solutions is 
nontrivial, i.e., not made only from zero warnings. We use the whitening core in the 
definition instead of the real set of frozen variables, because it does not depend on the 
definition of clusters and it has much smaller finite size effects. The existence of such a 
frozen phase was proven in the thermodynamical limit for K > 9 of the i^-SAT near to 
the satisfiability threshold in |ART06] . 
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In order to determine the freezing transition we start with a 3-SAT formula of iV 
variables and all possible clauses, and remove the clauses one by one independently at 
randouQ. We mark the number of clauses M s where the formula becomes satisfiable as 
well as the number of clauses Mf < M s where at least one solution starts to have a 
trivial whitening core. We repeat .B-times (B = 2 • 10 4 in fig. |4.ip and compute the 
probabilities that a formula of M clauses is satisfiable P s (a,N), and unfrozen Pf(a,N) 
respectively. Due to the memory limitation we could treat only instances which have 
less than 5 • 10 7 solutions which limits us to system sizes N < 100. The results for the 
satisfiability threshold are shown in fig. 11.31 and are consistent with previous studies in 
|KS94l lMZK+99b[[MZK+99a] . The probability of being unfrozen, P f (a, N), is shown in 
fig. O 
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Figure 4.1: Left: Probability that there exists an unfrozen solution as a function of the 
constraint density a for different system sizes. The clustering [ZDEB-4] and satisfiability 
[MPZ02J transitions marked for comparison. Right: A 1:20 zoom on the critical (crossing) 
point, our estimate for the freezing transition is aj = 4.254 ± 0.009. The curves are 
cubic fits in the interval a G (4,4.4). The arrows represent estimates of the limits 
of performance of the best known local search ASAT [AA06] and survey propagation 
jPar03l ICFMZ05] algorithms. 

It is tempting to perform a scaling analysis as has been done in [KSM lMZK+99bj 
lMZK + 99a] for the satisfiability threshold. The critical exponent related to the width of 
the scaling window was defined via rescaling of the constraint density a as N l l Ua [1 — 
a/a s (N)]. Note, however, that the estimate v s = 1.5 ± 0.1 for 3-SAT provided in 
[MZK + 99a] is not asymptotically correct. It was proven in [Wil02j that v 8 > 2. In- 
deed, it was shown numerically in [LRTZ01J that a crossover exists at sizes of order 
iV w 10 4 in the related XOR-SAT problem. A similar situation happens for the scaling 
of the freezing transition, Pj(a,N), as the proof of |Wil02] applies also here[E It would 
be interesting to investigate the scaling behaviour on an ensemble of instances where the 
results of }Wil02] do not apply (e.g. graphs without leaves). However, we concentrate 
instead on the estimation of the critical point, which we do not expect to be influenced by 
the crossover in the scaling. We are in a much more convenient situation for the freezing 
transition than for the satisfiability one. The crossing point between functions Pf(a,N) 

practice we do not start with all the clauses, but as many that in all the repetitions of this 
procedure the initial instance is unsatisfiable. 

2 Theorem 1 of [Wil02 applies to the freezing property where the bystander are clauses containing 
two leaves. 
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for different system sizes seems to depend very little on N, while for the satisfiability 
transition it depends very strongly on N, compare the zooms in figs. If .31 and 14. 1 1 
We determine the value of the freezing transition in random 3-SAT as 

a f = 4.254 ±0.009, (4.1) 

which is very near but seems separated from the satisfiability threshold a s = 4.267 
[MZ02, MMZ06J. In any case the frozen phase in 3-SAT is very narrow, that is in contrast 
with the situation in K > 9 SAT where it covers at least 1/5 of the large clustered phase 
[ART06j . 

4.2 Cavity approach to frozen variables 

In this section we present how to describe the frozen variables within the 1RSB cavity 
solution. We illustrate the results on an example of the random graph coloring where 
properties of frozen variables were studied in detail for the first time [ZDEB-5]. 

The energetic 1RSB (survey propagation), sec. I1.6H1.71 aims to count the total number 
of frozen clusters. More precisely, it counts the total number of fixed points of the warning 
propagation (11.351) . It can be used to locate the satisfiability threshold or to design survey 
propagation based solvers |MPZ02t IMZ02J . However, as we understood in chapter [2, by 
neglecting the soft clusters we cannot locate the clustering transition. In chapter [3] we 
defined the dominant clusters, i.e., those which cover almost all solutions. A natural 
question arises immediately: Are the dominant clusters frozen or soft? In order to answer 
the general entropic 1RSB equations (12.24112.281) have to be analyzed. 

4.2.1 Frozen variables in the entropic 1RSB equations 

We remind that in the 1RSB solution of the graph coloring problem the components 
of the messages (called also the cavity fields) ip^i are the probabilities that in a given 
cluster the node i takes the color Sj when the constraint on the edge (ij) is not present. 
The belief propagation equations, (I1.16P in general, (12.51) in coloring, then define the 
consistency rules between the field and fields incoming to i from the other variables 
than j. In the zero temperature limit we can classify fields in the following two 
categories: 

(i) The hard (frozen) field corresponds to the case when all components of ■0*~ >J are 
strictly zero except the one for color s. This means that in the absence of edge (ij), 
variable i takes color s in all the solutions from the cluster in question. 

(ii) The soft field corresponds to the case when more than one component of Vs"*- 7 is 
nonzero. The variable i is thus not frozen in the absence of edge (ij), and the colors 
of all the nonzero components are allowed. 

This distinction is also meaningful for the full probabilities (11.181) . By definition, the 
variable i is frozen in the cluster if and only if ip l is a hard field. 

It is important to stress that some of the soft fields on a given instance of the problem 
might be very small. Some of them might even scale like e~ N . We insist on classifying 
those as the soft fields because they cannot create real contradictions. This subtle dis- 
tinction becomes important mainly in the implementation of the population dynamics 
algorithm, see appendix [0 
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The distribution of fields over clusters P l ~^ J (■0* ->:? ) ( 12.24ft . which is the "order param- 
eter" of the 1RSB equation, can be decomposed into the hard-field part of a weight r/*^- 7 
and the soft-field part of a weight rf^ 3 = 1 — Y^s=i 

q 

pi^j (^-i) = ^'I^^'frozen into s) + rfr j P^ • (4.2) 



Hard fields in the simplest case, m = — First, we derive equations for the 
hard fields when the parameter m = in (I2.24H . This will, in fact, lead to the survey 
propagation equations, for coloring originally derived in [M PWZ021 lBMP + 03] from the 
energetic 1RSB method (II .6p . For simplicity we write the most general form only for the 
3-coloring. 

We plug (14.21) into eq. ( 12.241) . The reweighting factor (^Z l ^) m at m = is either 
equal to zero, when the arriving fields are hard and contradictory, or equal to one. This 
is the origin of a significant simplification. The outcoming field might be frozen in 
direction s if and only if for every other color r ^ s there is at least one incoming field 
frozen to the color r. The update of probability rf^i that a field is frozen in direction s 
is for the 3-coloring written as 

_ Tlkei-jO- ~ Vs^ 1 ) ~ Yj P1 Ls rifcgi-j(^o~ >t + Vp~* 1 ) + l~Ifcei-j VoT 1 ^ ^ 

In the numerator there is a telescopic sum counting the probability that color s and 
only color s is not forbidden by the incoming fields. In the denominator there is the 
normalization, i.e., the telescopic sum counting the probability that there is at least one 
color which is not forbidden. The crucial observation is that at m = the self-consistent 
equations for r\ do not depend on the soft-fields distribution ^P^^)- 

If we do not aim at finding of a proper coloring on a single graph but just at computing 
of the complexity function and similar quantities, we can further simplify eq. (14.31) by 
imposing the color symmetry. Indeed, the probability that in a given cluster a field is 
frozen in the direction of a color s has to be independent of s. Then H4.3|) becomes, now 
for general number of colors q: 



^, _ sa(-i)'C7 1 )rwi-c + i),M (4 4) 
ESG(-i) , Gyn* a - j [i-(f+i)i*- rf i ' 



We remind that since dY>(s)/ds = —m (I2.28p . the value m = corresponds to the 
point s where the function E(s) has a zero slope. If a nontrivial solution of ( 14. 3ft exists, 
then E(s)| m= o is the maximum of the curve E(s). And if the 1RSB solution for clusters 
at m = is correct then it is counting the total log-number of clusters of size s, which is 
due to the exponential dependence also the total log-number of all clusters, regardless of 
their size. 



Frozen variables at general m, generalized SP — Let us compute how the fraction 
of hard fields r] evolves after one iteration of equation (I2.24p at a general value of m. There 
are two steps in each iteration of ( 12.241) . In the first step, rj iterates via eq. ( 14.4ft . In the 
second step we re- weight the fields. Writing P^ ard (Z) the — unknown — distribution of 
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the reweightings Z m for the hard fields, one gets 

_ w({r] k ^ 1 }) C ■■ hard , ^ , • ^ m _ w({rj k ^ 1 }) 



J dz ^ p^z^) (Z^) m = W{{ ^ }) (^')ha rd • (4.5) 



A similar equation can formally be written for the soft fields 

l-rf-'^ ^P (Zl-'U,- («) 

Writing explicitly the normalization A/"*~* J ', we finally obtain the generalized survey prop- 
agation equations: 

rj^ = : (4 . 7) 

gu^lr/^ 4 }) + [1 — gw({^' c ^ 4 })] r(m, {f? fc_n }) 

where r is the ratio of average reweighting factors of the soft and hard fields 

Km, {^}) = • ( 48 ) 

/hard 

In order to do this recursion, the only nontrivial information needed is the ratio r between 
soft- and hard-field average reweightings, which depends on the full distribution of soft 
fields Pl~£l (ip 1 ^ 3 ) . Eq. (14. 7j) is easy to use in the population dynamics and allows to 
compute the fraction of frozen variables in typical clusters of a given size (for a given 
value m). 

There are two cases where eq. ( 14 .TP simplifies so that the hard-field recursion becomes 
independent from the soft-field distribution. The first case is, of course, m = 0. Then 
r = 1 independently of the edge (ij), and the equation reduces to the original SP. The 
second case arises for m = 1, where the eq. (14 .7p can be written as the equation for the 
naive reconstruction (12.4ft . The probability that a variables is frozen at m — 1 is the 
same at the probability that leaves (far away variables) determine uniquely the root in 
the reconstruction problem, see sec. 12.1.11 

Frozen variables and minimal rearrangements — Montanari and Semerjian [MS05, 
ISem08j developed a very interesting connection between frozen variables and the so-called 
minimal rearrangements. Given a CSP instance, one of its solutions {s{} and a variable 
i, find the nearest solution to {sj} where the values of the variable i is changed to ^ Sj. 
The set of variables on which these two solutions differ is called the minimal rearrange- 
ment. It was shown in |Sem0 8j that the size of the average (over variables i, the solution 
{s^, and the graph ensemble) minimal rearrangement diverges at the rigidity transition 
(when almost all the dominant clusters become frozen). Indeed, the cavity approach to 
minimal rearrangements leads to equations analogous to those for frozen variables. Part 
of the reasoning is the following [SAO05] : Consider a solution of a i^-SAT formula and 
a variable i from its whitening core. By flipping the variable % at least one neighbouring 
constraint a is made unsatisfied, otherwise the variable would not be in the whitening 
core. All variables contained in a are also in the whitening core, thus one of them has 
to be flipped in order to satisfy this constraint. There have to be a chain of flips which 
can be finished only by closing a loop. The length of the shortest loop going through a 
typical variable is of order log N. Thus a diverging number of changes is needed to find 
another solution. Hence the connection between frozen variables and rearrangements is: 
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• If the variable i is frozen in the cluster to which the solution {s^ belongs, then in 
order to change the value of i one has to find a solution from a different cluster, 
thus at an extensive Hamming distance. 

• If the variable i is not frozen in the cluster to which the solution {sj} belongs, then 
the best rearrangement will probably also lie within that cluster and the Hamming 
distance is finite. 

Many more results about rearrangements can be found in [Sem08j . they shed light 
on the onset of frozen variables. An exciting possibility is that the cavity equations 
for rearrangements might be useful in incremental algorithms for CSPs, like the one of 
|KK07j . 



4.2.2 The phase transitions: Rigidity and Freezing 

A natural question is: "In which clusters are the hard fields present?" Or more in the 
terms of the 1RSB solutions: "When does eq. ( 14.7ft have a nontrivial solution rj > 0?" 
We answer this question in one of the simplest cases, that is for the coloring of random 
regular graphs of connectivity c = k + 1. In tree- like regular graphs the neighbourhood of 
each node looks identical, thus also the distribution P 1 ^ (ip 1 ^) is the same for every edge 
(ij). Moreover we search for a color-symmetric solution [ZDEB-5], that is t] s = r] r = r\ 

for all s,r 6 {1 q}. The function w({rj}) in the ensemble of random regular graphs 

simplifies to 

SS^DTT'nw + D^ (49) 
E& 1 (-i)'(>?i)[ 1 -(' + 1 )if ' 

First notice that in order to constrain a variable into one color, i.e., create a hard field, 
one needs at least q — 1 incoming fields that forbids all the other colors. It means that 
the function w({r]}) defined in eq. (14. 9p is identically zero for k < q — 1 and might be 
non-zero only for k > q — 1, where k is the number of incoming fields. 

The equation (14. 7p also simplifies on a regular graph and rj follows a self-consistent 
relation 

77 = wirj) — — . N1 . — -, (4.10) 

qw{r]) + [1 — qw[ri)\ r(m) 

where r(m) is the average of the reweighting of the soft fields divided by the average of 
the reweighting of the frozen fields (14.71) . The function r(m) is in general not easy to 
compute, the population dynamics is needed for that. Several properties are, however 
known: 

r — > when m — > — oo , (4.11a) 
r — > oo when m — > oo , (4.11b) 

and r(m) is a monotonous function of m. Moreover, for the internal entropy of clusters 
s(m) — > when m — > — oo, and s(m) — > oo when m — > oo, and s(m) is also a monotonous 
function. We thus solve eq. (14.101) for every possible ratio r. For all k > q — 1 we compute 
the solution rj(r). The result is shown in fig. 14.21 for the 3- and 4-coloring of random 
regular graphs. 

There is a discontinuous phase transition: For r < r r eq. ( 14.101) has a solution with a 
large fraction of frozen fields, 7] > 0, whereas for r < r r the only solution is rj — 0. Note 
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Figure 4.2: The lines are solutions of eq. (14.101) and give the fraction qr] of hard fields 
for a given value of ratio r = (Z^) so f t / (^^Ohard for g = 3 (left) and q = 4 (right) in 
regular random graphs. There is a critical value of the ratio r r (red point) beyond which 
only the trivial solution 77 = exists. Note that the solutions corresponding to m = 
(green square) and m = 1 (blue triangle) only exist for a connectivity large enough. 

that the index r stands for "rigidity". In terms of the parameter m, the critical value is 
r(m r ) = r r . In terms of the internal entropy of clusters s(m r ) = s r . The interpretation 
is the following: 

• Clusters of internal entropy s < s r are almost all frozen, and the fraction of frozen 
variable's they contain is quite large. 

• Clusters of internal entropy s > s r are almost all soft, meaning the fraction of frozen 
variables is zero. 

When we change the average constraint density there are at least three interesting 
phase transitions related to frozen variables. Fig. 14.31 sketches the difference between the 
phases they separate. Recall that s* is the internal entropy of the dominant clusters, and 
s max the internal entropy of the largest clusters S(s max ) = 0. 

• The rigidity transition, c r , at which s* = s r , separates a phase where a typical 
dominant cluster is almost surely not frozen from a phase where a typical dominant 
cluster is almost surely frozen. 

• The total rigidity transition, c tr , at which s max = s r , when almost all clusters of 
every size become frozen. 

• The freezing transition, Cf, separates phase where exponentially many unfrozen 
cluster exists from a phase where such clusters almost surely do not exisJE 

In general it have to be c r < Q r < c/. The relation between the rigidity and total 
rigidity transition is easily obtained from the 1RSB solution. It is thus known that in the 
g-coloring of Erdos-Renyi graphs c r = c tr if and only if q < 8, in i^-SAT if and only if 
K < 5. For larger q or K the rigidity transition is given by the onset of frozen variables 
in clusters corresponding to m = 1, this is equivalent to the naive reconstruction (12.41) . 

3 Note that what is called freezing transition in |Sem08| or in sec. IV. C of ;MRTS08 is in fact what 
we define as the rigidity transition, in agreement with [ZDEB-5]. 
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Figure 4.3: A pictorial sketch of the complexity function of clusters of a given size. Cyan- 
blue is the complexity of the frozen clusters, magenta of the soft clusters. The total 
complexity is the envelope, which can be calculated from the entropic 1RSB solution. 
The black point marks the dominating clusters. Left: In the rigid phase almost all the 
dominant clusters are frozen, but clusters corresponding to larger entropy might be mostly 
soft. Middle: In the totally rigid phase almost all clusters of all sizes are frozen, but there 
still might be exponentially many of soft clusters. Right: The frozen phase where soft 
clusters almost surely do not exist. 

The relation between the total rigidity transition and the freezing is less known. There 
are only few studies for the freezing transition in random iT-SAT. The first one is the 
one of (ART06] where they prove that for every K > 9 the freezing transition is strictly 
smaller than the satisfiability one Cf < c s . In the large K limit they showed that the 
frozen phase covers a finite fraction (at least 20%) of the satisfiable region. The second 
study [MS07J gives a rigorous upper bound on the freezing transition in 3-SAT «/ < 4.453, 
which is slightly better than the best known upper bound on the satisfiability transition 
in 3-SAT jDBMOO] . The third study is numerical [ZDEB-10], presented in fig. EJ It 
shows that in 3-SAT the frozen phase is tiny, about 0.3% of the satisfiable region. 

It is not known if the total rigidity transition coincides with the freezing transition. 
The entropic cavity method describes a typical but not every cluster of a given size. 
A generalization of the 1RSB equations which would count only the number of soft 
cluster would answer this question. 

To summarize the description of the freezing of variables and clusters in the canonical 
constraint satisfaction problems, like g-coloring or .fT-satisfiability, is both numerically 
and conceptually involved task. Moreover in the experimentally feasible range of q and 
K the frozen phase is tiny. Thus conclusive statements about the connection between 
the freezing and the computational hardness are difficult to make. In the next section 
we introduce the so-called locked constraint satisfaction problems where the situation is 
much more transparent. 



4.3 Point like clusters: The locked problems 

In order to get a better understanding of the frozen phase we introduce the so-called 
locked constraint satisfaction problems [ZDEB-9]. In these problems the whole clustered 
phase is at the same time frozen, this is because in the locked problems all the clusters 
contain only one solution. 
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4.3.1 Definition 

A locked constraint satisfaction problem is made of iV variables and M locked constraints 
in such a way that every variable is present in at least two constraints. A constraint 
consisting of K > variables is locked if and only if for every satisfying assignment 
of variables changing the value of any (but only one) variable makes the assignment 
unsatisfying. 

A locked constraint of K variables has the property that if (K — 1) variables are 
assigned then either the constraint cannot be satisfied by any value of the last variable 
or there is only one value of the last variable which makes the constraint satisfied. All 
the uniquely extendible constraints [Con04l ICM04] are locked, XOR-SAT being the most 
common example. 1-in-K SAT (exact cover) constraint |GJ79j is another common ex- 
ample. On the other hand, the most studied constraint satisfaction problems K-SAT or 
graph g-coloring (q > 2) are not made of locked constraints. 

The second important part of the definition of locked constraint satisfaction problems 
is the requirement that every variable is present in at least two constraints, i.e., leaves are 
absent. An important property follows: In order to change a satisfying assignment into 
a different satisfying assignment at least a closed loop of variables have to be changed. If 
leaves would be allowed changing a path connecting two leaves might be sufficient. 

It seems to us that all the random locked constraint satisfaction problems should 
behave in the way we describe in the following. We, however, investigated in detail only 
a subclass of the locked problems called locked occupation problems (LOP). Occupation 
constraint satisfaction problem is defined as a problem with binary variables (0-empty, 
1-occupied) where each constraint containing K variables is a function of how many of the 
K variables are occupied. A constraint of the occupation CSP can thus be characterized 
via a (K + l)-component vector A, e {0, 1}, i G 0, . . . , K. A constraint is satisfied 
(resp. violated) if it contains r occupied variables where r is such that A r — 1 (resp. 
A r = 0). For example A = (0, 1,0,0) corresponds to the positive l-in-3 SAT [ZDEB-3], 
A = (0, 1, 1, 0) is bicoloring [CNRTZ03 ]. A = (0, 1, 0, 1, 0) is 4-odd parity check (4-XOR- 
SAT without negations) [MRTZ03] . 

An occupation problem is locked if all the variables are connected to at least two 
constraints and the vector A is such that AiA i+ i — for alH = 0, . . . , K — 1. We study 
the random ensembles of LOPs where all constraints are identical and the variable degree 
is either fixed of distributed according to a truncated Poissonian law (II. 6p . 

4.3.2 The replica symmetric solution 



The replica symmetric cavity equations, belief propagation fll.16am.16bj) . for the occu- 
pation problems read 



{sj} j&da—i 

^=ra n ( 4 - i2b ) 



b€dj—a 



where ip^ 1 is the probability that the constraint a is satisfied conditioned that the 
value of the variable % is s», and xi~* a is t ne probability that variable j takes value Sj 
conditioned that the constraint a was removed from the graph. The normalizations Z 
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have the meaning of the partition function contributions. The replica symmetric entropy 
s is a zero temperature limit of f 1 1 . 2 1) 

a i 

where the contributions ^ a+9a (resp. Z l ) are the exponentials of the entropy shifts when 
the node a and its neighbours (resp. the node i) is added ( 11. 19atfT7l9b[) 

Z a + 8a = - 1) II ( II ) > ( 4 - 14a ) 

{si} i&a \bei~a / 

zi = n^+n^r*- ( 4 - i4b ) 



Solving eqs. (j4.12ati4.12bp means finding their fixed points. A crucial property of the 
locked problems it that if {sj} is one of the solutions then 

il%« = l, V^ = 0, (4.15a) 
xr a = l, Xli: = (4.15b) 



is a fixed point of eqs. ( I4.12atl4~12bp . The corresponding entropy is then zero, as Z l 



Z a+da = 1 for all i, a. In the derivation of [MM08J fixed points of the belief propagation 
equations correspond to clusters. Thus in the locked problems every solution corresponds 
to a cluster. 

In the satisfiable phase there exist exponentially many solutions (i.e., clusters), thus 
the iterative fixed point of BP equations (I4.12atl4.12bp obtained from a random initial- 
ization gives an asymptotically exact value for the total entropy. And the satisfiability 
threshold coincides with the condensation transition, described in chap. [31 Furthermore, 
as each cluster contains only one solution the clustered phase is automatically frozen 
according to the definition in sec. 14.2.21 Interestingly, part of the satisfiable phase is 
only "fake clustered" meaning that at infinitesimally small temperature there is a single 
fixed point of the BP equations. This has been discussed e.g. in the context of the per- 
fect matchings in [ZDEB-1]. A general discussion and proper definition of the clustering 
transition in the locked problems follows in sec. 14.3.31 

Iterative fixed point of eqs. ( I4.12all4~14bp averaged over the graph ensemble is in 
general found via the population dynamics technique, see appendix [El Note that the 
sum over {sj} in (14.12al) can be computed iteratively in (K — l) 2 steps instead of the 
naive 2 K ~ l steps. Moreover, on the regular graphs ensemble or for some of the symmetric 
locked problems, such that = for all i — 0, . . . , K, the solutions is factorized. In 
the factorized solution the messages x % ^ a i ip a ^ 1 are independent of the edge (ia) and the 
population dynamics is thus not needed. 

• For the regular graph ensemble where each variable is present in L constraints the 
factorized solution is 

* = ^Er; 1 )^ 1 -*- 1 -'' (4.16a) 

A r = l ^ ' 

hi E (V>f-*tf- I,( *- 1 - r) , < 4 - 16b > 

Ar+l=l ^ ' 



A r =l 

1 
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and the entropy is 



'reg 



K l ° g 



K 



n 



l)r^(L~l)(K-r) 



(L — 1) log [ipQ + ipi] . (4.17) 



For the symmetric locked problems where the symmetry is not spontaneously broken 
the solution is also factorized. We call these the balanced locked problems. The BP 
solution is ipi — i/jq — 1/2 and the corresponding entropy 



Ssym(0 = log 2 + 



2"<±6(A r -l)( K ) 



(4.18) 



where I is the average degree of variables. Notably, this result for the entropy can 
be proven rigorously by computing the first and second moment of the partition 
sum, i.e., (Z), (Z 2 ), and using the Chebyshev's inequality. The exact value of the 
satisfiability threshold is then given by s sym (l s ) = 0. This itself is a remarkable 
result, because so far the exact threshold was computed in only a handful of the 
sparse NP-complete CSPs. As far as we know only in the 1-m-K SAT [ACI MOlJ 
and [ZDEB-3], the 2+p-SAT |MZK+99allAKKK01] and the (3,4)-UE-CSP |UM04j . 
We dedicate the appendix [B] to this computation. 

The replica symmetric solution might be incorrect if long range correlations are present 
in the system, as we discussed in detail in chap. 121 A sufficient condition for its correctness 
is the decay of the point-to-set correlations, which we will discuss in the next section, again 
in context of the reconstruction problem. A necessary condition for the RS solution to 
be correct is the non- divergence of the spin glass susceptibility, which can be investigated 
in several equivalent ways, as described in appendix O The result for all the locked 
problems we investigated is that the phase where the entropy ( 14. 13|) is positive is always 
RS stable, whereas part of the phase where the entropy (14.131) is negative might be RS 
unstable (depending on the parameters and the vector A). 



4.3.3 Small noise reconstruction 

It is immediate to observe that reconstruction as we defined it in sec. 12.1.11 is always 
possible for the locked problems. Indeed, if we know K — 1 out of K variables around a 
constraint the last one is given uniquely (no contradiction is possible as we broadcasted 
a solution). This is related to the fact that at least one closed loop has to be flipped to 
go from one solution of a given instance of a locked problem to another solution. Typical 
length of such a minimal loop is of order logiV. For very low connectivities, and at 
infinitesimally low temperature, the BP equations will have a unique fixed point, there 
the zero temperature logiV clustering is "fake" and will not have a crucial influence on 
the dynamics and other properties of interest. 

Thus for the locked problem it is useful to modify the definition of the clustering 
transition presented in chap. [2j In order to do that we need to introduce the small noise 
( SN) reconstruction. Construct an infinite tree hyper-graph, assign a value 1 or to its 
root and iteratively assign its offsprings uniformly at random but in such a way that the 
constraints are satisfied (constraints play the role of noiseless channels). At the end of the 
procedure forget the values of all variables in the bulk but also of an infinitesimal fraction 
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e of leaves. If the remaining 1 — e leaves contain some information about the original value 
on the root then we say that the small noise reconstruction is possible, if they do not the 
small noise reconstruction is not possible. The phase where the SN reconstruction is not 
possible is then only "fake clustered" and is more similar to the liquid phase. Whereas 
the phase where the SN reconstruction is possible has all the properties of the clustered 
phase, except that each of the clusters contains only one configuration^. 

All the equations we derived in sec. 12.1.11 for the reconstruction apply also for the SN 
reconstruction. Except the specification of the initial conditions (12.111) which for the SN 
reconstruction is instead 



where e < 1. The second term accounts for the fraction of leaves on which the value of the 
variable has been forgotten. The fixed point of the 1RSB equation ( 12.241) is then either 
trivial (corresponding to the replica symmetric solution) or nontrivial describing solutions 
as an ensemble of totally frozen clusters. This has several interesting consequences: The 
threshold for the naive SN reconstruction (i.e., the one taking into account only the 
frozen variables) coincide with the true threshold for SN reconstruction. The solution 
of the 1RSB equation ( 12. 24ft in the locked problem does not depend on the value of the 
parameter m. 

A general form of the 1RSB equations at m = 1 for occupation problems is derived 
in appendix |A] First we consider only problems where the replica symmetric solution 
is factorized. We define fi\ (resp. /xq) as the probability that a variable which in the 
broadcasting had value 1 (resp. 0) is uniquely determined by the boundary conditions. 
Based on the general eq. ( 1A.10I) . we derive self-consistent equations for n\, /i on regular 
graphs ensemble of connectivity of variables L: 



where I = L — 1, k = K — 1. The indices si, So in the second sum of both equations 
are the largest possible but such that s\ < r, so < K — 1 — r, and YllLo^r-s = 0, 
Sslo A r +i+s = 0. The values ip , ipi are the fixed point of eqs. (j4.16aH4.16bl) . and Z Teg is 
the corresponding normalization. These lengthy equations have in fact a simple meaning. 
The first sum is over the possible numbers of occupied variables on the descendants in 
the broadcasting. The sums over s is over the number of variables which were not implied 
by at least one constraint but still such that the set of incoming implied variables implies 
the outcoming value. The term 1 — (1 — fi) 1 is the probability that at least one constraint 
implies the variable, (1 — fi) 1 is the probability that none of the constraints implies the 
variable. 



p init w = ^ w - s ) + s$ - + es{4> - h s&i - h , 



(4.19) 




) 



(4.20b) 



(4.20a) 



4 Note that a rigorous study of a related robust reconstruction exists JM04 . In robust reconstruction, 
however, one allows e to be arbitrarily near to one. 
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The second case where the BP equations are factorized are the balanced locked prob- 
lems. That is LOPs with symmetric vector A where the symmetry is not spontaneously 
broken. Then ip — ipi — 1/2 and thus also fi = Hi = /i. For the ensemble of graphs 
with truncated Poissonian degree distribution of coefficient c we derive from flA.lOp 



1 — e~ CM \ k ~ s ( e~ c/J 



1-e- 



(4.21) 



where k = K — 1, and #a = A r 1=1 + A r =i © anc ^ ^ ne varue s i s 5 as before, the 
number of descendants which were not directly implied. 

In both these cases, there are two solutions to eqs. (I4.20ati4.20bl) and f)4.2ip . One 
is /i = and the other \i = 1. The small noise reconstruction is investigated by the 
iterative stability of the solution /i = 1. If it is stable then the SN reconstruction is 
possible, all variables are almost surely directly implied. If it is not stable then the only 
other solution is /x = 0. Few observations are immediate, for example if L > 3 then the 
solution Hi = fi = 1 of fl4.20aH4.20bj) is always iteratively stable. Iterative stability of 
f!4.2ip gives for the balanced locked problems, marked by * in tab. 14.11 



K - 1 



J2«S 2 5(A r+ i-l)5(A r 



S(Ar) (V) 



i) (V 1 ) 



(4.22) 



4.3.4 Clustering transition in the locked problems 



In the locked problem where the replica symmetric solution is not factorized there is 
another equivalent way to locate the clustering transition, which is simpler than solving 
eq. ( 1A.10I) . It is the investigation of the iterative stability of the nontrivial fixed point of 
the survey propagation. In LOPs the survey propagation equations consist of eqs. ( I1.4ip 
and 



1 
1 

J\[a-*i 



{r-j} j£a~i 

E^(^» n p*r 

{r-j} jea-i 
{r-j} jea-i 



(4.23a) 
(4.23b) 
(4.23c) 



where the indexes rj e {1,-1,0}, M a ^ % is the normalization constant. The Ci/C_i 
(resp. Co) takes values 1 if and only if the incoming set of {rj} forces the variable i to 
be occupied/empty (resp. let the variable % free), in all other cases the C's are zero. Let 
us call si, s_x, s the number of indexes 1, —1, in the set {rj} then 



Ci = 1 if and only if A Sl 



+s +l 



1 and A, 



l+n 



for all n = . . . sq; 



C_i = 1 if and only if A S1 = 1 and A Sl+ i +n — for all n — . . . s ; 

Co = 1 if and only if there exists m, n — . . . sq such that A Sl+n = A Sl+m+ i = 1. 
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A 


name 


^ S 




Co 






0100 


l-in-3 SAT 


3 


685(3) 


946(4) 


2 256(3) 


2 368(4) 


01000 


l-in-4 SAT 


3 


1.108(3) 


1.541(4) 


2.442(3) 


2.657(4) 


00100* 


2-in-4 SAT 


3 


1.256 


1.853 


2.513 


2.827 


01010* 


4-odd-PC 


5 


1.904 


3.594 


2.856 


4 


010000 

V J _L V J V 7 V J V J 


l-in-5 SAT 


3 


1.419(3) 


1 982(6) 


2 594(3) 


2 901(6) 


001000 

V 7 V 7 -L V 7 V J V J 


2-in-5 SAT 


4 


1 604(3) 

-L . V7 V7 J. I ' 7 / 


2 439(6) 


2 690(3) 


3 180(6) 


010100 


l-or-3-in-5 SAT 


5 


2 261(3) 


4 482(6) 


3 068(3) 


4.724(6) 


010010 


l-or-4-in-5 SAT 


4 


1.035(3) 


2.399(6) 


2.408(3) 


3.155(6) 


0100000 


l-in-6 SAT 


3 


1.666(3) 


2.332(4) 


2.723(3) 


3.113(4) 


0101000 


l-or-3-in-6 SAT 


6 


2 519(3) 


5 123(6) 


3 232(3) 


5 285(6) 


0100100 


l_ or -4-in-6 SAT 


4 


1 646(3) 


3 366(6) 


2 712(3) 


3 827(6) 


0100010 


l-or-5-in-6 SAT 


4 


1 594(3) 


2 404(6) 


2 685(3) 


3 158(6) 


0010000 


2-in-6 SAT 


4 


1.868(3) 


2.885(4) 


2.835(3) 


3.479(4) 


0010100* 


2-or-4-in-6 SAT 


6 


2.561 


5.349 


3.260 


5.489 


0001000* 


3-in-6 SAT 


4 


1.904 


3.023 


2.856 


3.576 


0101010* 


6-odd-PC 


7 


2.660 


5.903 


3.325 


6 



Table 4.1: The locked cases of the occupation CSPs for K < 6 (cases with a trivial 
ferromagnetic solution are omitted). In the regular graphs ensemble the phase is clustered 
for L > Ld = 3, and unsatisfiable for L > L s . Values c are the critical parameters of the 
truncated Poissonian ensemble fll.6|) . the corresponding average connectivities I are given 
via eq. (11. 7ft . All these problems are RS stable at least up to the satisfiability threshold. 
For the balanced cases, marked as *, the dynamical threshold follows from (14.211) . and 
the satisfiability threshold, which can be computed rigorously, app. from (14.181) . 



The SP equations in LOPs have two different fixed points: 

• The trivial one: = p l Q~* a = 1, q^ 1 = p l {^ a = q°i^ 1 = p % Z[ a = for all edges (ai). 

• The BP-like one: q^ 1 = p}^ a = 0, q a ^ 1 = ?p a ^\ p % ^ a = x l_>a for ah edges (ai), 
where ip and \ is the solution of the BP equations (I4.12all4.12bl) . 

The small noise reconstruction is then investigated, using the population dynamics, from 
the iterative stability of the BP-like fixed point. If it is stable then the SN reconstruction 
is possible and the phase is clustered. If it is not stable then we are in the liquid phase. Of 
course, this approach gives the same critical connectivity l& as the previous one, because 
for the locked problems the solutions of the 1RSB equation (I2.24p is independent of the 
parameter m. 

We remind at this point that in a general CSP, where the sizes of clusters fluctuate, 
the SP equations are not related to the reconstruction problem, more technically said the 
1RSB solutions at m = and at m = 1 are different. The solution of the locked problems 
is sometimes called frozen 1RSB |MMR041 |MMR05| . 



4.4 Freezing - The reason for hardness? 



We describe several strong evidences that it is hard to find frozen solutions. We also 
give several arguments for why it is so. However, the precise mechanism stays an open 
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question and strictly speaking the freezing of variables might just be going along with 
a true yet unknown reason. Or even there might be an algorithm which is able to find 
the frozen solutions efficiently waiting for a discovery. But in any case, we show that 
freezing of variables is an important new aspect in the search of the origin of the average 
computational hardness. 

4.4.1 Always a trivial whitening core 

Several studies of the random 3-SAT problem [ MMW071 lBZ04l ISAU05] showed that 
all known algorithms on large instances systematically find only solutions with a trivial 
whitening core (defined in sec. I4.1.1H . On small instances of the problem solutions with 
a nontrivial whitening core can be found as observed by several authors, and studied 
systematically in sec. 14. 1.31 

For solutions found by the stochastic local search algorithms, see appendix [Fj this 
observation is reasonable, as argued already in |SAO05j . Consider that a stochastic local 
search finds a configuration which is not a solution, but its whitening core is nontrivial. 
Then a diverging number of variables have to be rearranged in order to satisfy one of 
the unsatisfied constraints [Sem08] . In the clusters with a trivial whitening core the 
rearrangements are finite [Sem08j and thus stochastic local dynamics might be able to 
find them more easily. 

The fact of finding only the "white" solutions is, however, quite surprising for the 
survey propagation algorithm. The SP equations compute probabilities (over clusters) 
that a variables is frozen in a certain value. This information is then used in a deci- 
mation, reinforcement, etc. algorithms, see appendix [F] Thus SP is explicitly exploring 
the information about nontrivial whitening cores and in spite of that it finishes finding 
solutions with trivial whitening cores. 

A related, and rather surprising, result was shown in |DRZ08j . The authors considered 
the random bi-coloring problem in the rigid, but not frozen, phase. That is a phase where 
most solutions are frozen, but rare unfrozen ones still exist. They showed that belief 
propagation reinforcement solver, see appendix (Fj is in some cases able to find these 
exponentially rare, but unfrozen, solutions. 

We observed the same phenomena in one of the non-locked occupation problem A = 
(0110100), that is l-or-2-or-4-in-6 SAT. On regular factor graphs this problem is in the 
liquid phase for L < 6, in the rigid phase for 7 < L < 9, where almost all the solutions 
are frozen, and it is unsatisfiable for L > 10. In fig. 14.41 we show that belief propagation 
reinforcement finds almost always solutions for L — 8, but as the size of instances is 
growing the fraction of cases in which the solution is frozen goes to zero. 

We listed this paradox, that only the all-white solutions can be found, as one of the 
loose ends in sec. 11.81 The resolution we suggest here, and substantiate in the following, 
is that every known algorithm is able to find efficiently (in polynomial - but more often in 
experiments we mean linear or quadratic - time) only the unfrozen solutions. The frozen 
solutions are intrinsically hard to find and all the known algorithms have to run for an 
exponential time to find them. 

4.4.2 Incremental algorithms 

Adopted from |KK07] : Consider an instance of a constraint satisfaction problem of N 
variables and M constraints. Order randomly the set of constraints and remove all of 
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Figure 4.4: Algorithmical performance in the rigid phase of the l-or-2-or-4-in-6 SAT at 
L = 8. In red is the rate of success of the belief propagation reinforcement algorithms as 
a function of system size (out of 100 trials). The algorithm basically always succeeds to 
find a solution. In blue is the fraction of solutions which were frozen (had a nontrivial 
whitening core). Almost all solutions are frozen in this problem, yet it is algorithmically 
easier to find the rare unfrozen solutions, in particular in instances of larger size. 

them. Without constraints any configuration is a solutions. In each step: First, add 
back one of the constraints. Second, if needed rearrange the configuration in such a way 
that it satisfies the new and all the previous constraints. Repeat until there are some 
constraints left. We call such strategy the incremental algorithm for CSPs. And one can 
ask about its computational complexity. The way by which the rearrangement is found 
in the second step needs to be specified. But independently of this specification we know 
that if the new constraint connects frozen and contradictory variables then the size of 
the minimal rearrangement diverges [Sem08j . thus in the frozen phase the incremental 
algorithm have to be at best super-linear. 

Another understanding of the situation is gained by imagining the space of solutions 
at a given constraint density. As we are adding the constraints some solutions are disap- 
pearing and none are appearing. At the clustering transition the space of solutions splits 
into exponentially many clusters. As more constraints are added the clusters are becom- 
ing smaller, they may split into several smaller ones and some may completely disappear. 
However, only the frozen clusters can disappear, if a constraint is added between two 
frozen and contradictory variables. Note also that each frozen cluster will almost surely 
disappear before an infinitesimally small fraction of constraints is added. An unfrozen 
cluster, on the other hand, may only become smaller or split. Indeed, if a constraint is 
added any solution belonging to an unfrozen cluster may be rearranged in a finite num- 
ber of steps [Sem08] . The incremental algorithm in this setting works as a non- intelligent 
animal would be escaping from a rising ocean on a Pacific hilly island [KK07] . As the 
water starts to rise the animal would step away from it. As the water keeps rising at a 
point the animal would be blocked in one of the many smaller islands. This island will 
be getting smaller and smaller and it will disappear at a point and the animal will have 
to learn how to swim. But at this point there might still be many small higher island. 
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All of them will disappear eventually. For sure the animal will be in trouble before all 
the clusters (island) start to contain frozen variables. 

Moreover, if the sequence of constraints to be added is not known in advance there is 
no way to choose the best cluster, because which cluster is the best depends completely 
on the constraints to be added. This proves that no incremental algorithm is able to 
work in linear time in the frozen phase. On the other hand it was shown experimentally 
in |KK07j for the coloring problem that such algorithms work in linear time in part of 
the clustered (or even the condensed) phase. 

4.4.3 Freezing transition and the performance of SP in 3-SAT 



How does the freezing transition in 3-SAT, atf — 4.254 ± 0.009 fig. 14.11 compare to the 
performance of the best known random 3-SAT solver — the survey propagation? We are 
aware of two studied where the performance of SP is investigated systematically and with 
a reasonable precision, [Par03j and [CFMZ05J. 

In |Par03j the survey propagation decimation is studied. The SP fixed point is found 
on the decimated graph and the variable having the largest bias is fixed as long as the SP 
fixed point is nontrivial. When the SP fixed point becomes trivial the Walk-SAT algo- 
rithm finishes the search for a solutions. In |Par03j the residual complexity is measured 
on the partially decimated graph. It is observed that if the residual complexity becomes 
negative then solutions are never found, if on the other hand the residual complexity is 
positive just before the survey propagation fixed point become trivial then solutions are 
found. The value of complexity in the last step before the fixed point becomes trivial is 
extrapolated, fig. 2 of |Par03j for system size N = 3 ■ 10 5 , to zero at a constraint density 
a = 4.252 ± 0.003 (we estimated the error bar based on data from [Par03] ) . 

In [CFMZ05] the survey propagation reinforcement is studied. The rate of success is 
plotted as a function of the complexity function. From fig. 8 of |CFMZ05] it is estimated 
that SP reinforcement (more precisely its implementation presented in [CFMZ05J) finds 
solution in more than 50% of trials if S > 0.0013. The data do not really concentrate 
on this point, thus is is difficult to obtain a reliable error bar of this value, our educated 
guess is 0.0013 ± 0.0003 this would correspond to a constraint density a = 4.252 ± 0.004. 

The striking agreement between our value for the freezing transition and the perfor- 
mance limit of the survey propagation supports the suggestion that the frozen phase is 
hard for any known algorithm. The trouble for a better study of the frozen phase in 
3-SAT is its size, it covers only 0.3% of the satisfiable phase. In i^-SAT with large K the 
frozen phase becomes wider, but as K grows the constraint density of the satisfiability 
threshold grows like 2 K \ogK 1 empirical study thus becomes infeasible very fast. It is 
also not very easy to compute the freezing transition or to check if the 1RSB solution is 
correct in the frozen phase. Thus .fT-SAT (and g-coloring) are not very suitable problems 
for understanding better how exactly the freezing influences the search for a solution. 

4.4.4 Locked problems — New extremely challenging CSPs 

We introduced the locked problems to challenge the suggestion about hardness of the 
frozen phase [ZDEB-9]. It is rather easy to compute the freezing transition here, it 
coincides with the clustering transition Moreover, the frozen phase is wide, taking 
more than 50% of the satisfiable phase for some of the locked problems, see table 14.11 As 
in the locked problems every cluster consists of one solution, all the variables are frozen. 
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Consequently the replica symmetric approach describes correctly the phase diagram. 
From this point of view the locked problems seems extremely easy compared to if- SAT. 

On the other hand, experiments with the best known solvers of random CSPs show 
that the frozen phase of locked problems is very hard. And some of the very good 
solvers, e.g. the belief propagation based decimation, do not work at all even at the 
lowest connectivities (for an explanation see appendix [F]). 
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Figure 4.5: The probability of success of the BP-reinforcement (top) and the stochas- 
tic local search ASAT (bottom) plotted against the average connectivity for two of the 
locked occupation problems. The clustering transition is marked by a vertical line, the 
satisfiability threshold is l s = 4 for the 4-odd parity checks, and l s = 4.72 for the l-or-3- 
in-5 SAT. The challenging task is to design an algorithm which would work also in the 
clustered phase of the NP-complete locked problems. 

In fig. 14.51 we show the performance of the BP-reinforcement and the stochastic 
local search ASAT algorithms. Both the algorithms are described in appendix [Fj they 
are the best we were able to find for the locked CSPs. The greediness parameter in the 
stochastic local search ASAT we evaluated as the most optimal is p = 5.1CT 5 for the 
4-odd parity check, and p = 3.1(T 5 for the l-or-3-in-5 SAT. In the BP-reinforcement 
the optimal forcing parameter n changes slightly with the connectivity. For the l-or-3- 
in-5 SAT we used vr = 0.42 for 2.9 < 7 < 3.0 and tt = 0.43 for 3.0 < 7 < 3.2. For the 
4-odd parity checks we used it = 0.44 for 2.75 < 7 < 2.95. 

Of course, the parity check problem is an exceptional locked problem, as it is not 
NP-complete and can be solve via Gaussian elimination. However, our study shows that 
algorithms which do not use directly the linearity of the problem fail in the same way 
as they do in the NP-complete cases. Instances of the regular XOR-SAT indeed belong 
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between the hardest benchmarks for all the best known satisfiability solvers which do not 
explore linearity of the problem, see e.g. [HJKN06J. 



Fig. 14.51 puts in the evidence that in all the random locked problems the best known 
algorithms stop to be able to find solutions (in linear time) at the clustering transition. 
This supports the conjecture about freezing being relevant for algorithmical hardness. 
The locked problems are thus (at least until they are "unlocked") the new benchmarks 
of hard constraint satisfaction problems. 
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Chapter 5 

Coloring random graphs 



In the previous three chapters we developed tools for describing the structure of solution 
and the phase diagram of random constraint satisfaction problems. These tools were 
applied to the problem of coloring random graphs in a series of works [ZDEB-4, ZDEB-5, 
ZDEB-6, ZDEB-7]. In this section we summarize the results. 

5.1 Setting 

Coloring of a graph is an assignment of colors to the vertices of the graph such that 
two adjacent vertices do not have the same color. The question is if on a given graph 
a coloring with q colors exists. Fig. 15.11 gives an example of 3-coloring of a graphs with 
N = 22 vertices and M = 27 edges, the average connectivity is c = 2M/N « 2.45. 



It is immediate to realize that the g-coloring problem is equivalent to the question of 
determining if the ground-state energy of a Potts anti-ferromagnet on a random graph 
is zero or not [KS87J. Consider indeed a graph G = (V,E) defined by its vertices 
V = {1,...,N} and edges G E which connect pairs of vertices i,j G V; and 

the Hamiltonian 



With this choice there is no energy contribution for neighbours with different colors, but 
a positive contribution otherwise. The ground state energy is thus zero if and only if the 
graph is g-colorable. This transforms the coloring problem into a well-defined statistical 
physics model. 




Figure 5.1: Example of a proper 3-coloring of a small graph. 




(5.1) 
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Studies of coloring of sparse random graphs have a long history in mathematics and 
computer science, see [ZDEB-5] for some references. From the statistical physics perspec- 
tive it was first studied in |vMS02j . where the replica symmetric solution was worked out, 
and the replica symmetric stability was investigated numerically. Results were compared 
to Monte Carlo simulations and simulated annealing was used as a solver for coloring. The 
energetic 1RSB solution and the survey propagation algorithm for graph coloring were 
developed in [MPWZ021 lBMP+03j . Subsequently |KPW04| studied the stability of the 
1RSB solution and its large q limit. The entropic 1RSB solution was studies in [MPR05J 
for 3-coloring of Erdos-Renyi graphs. The entropic 1RSB solution was, however, fully 
exploited only in [ZDEB-4, ZDEB-5, ZDEB-6, ZDEB-7] and the resulting phase diagram 
is discussed here. 

5.2 Phase diagram 

Fig. 15.21 summarizes how does the structure of solutions of the coloring problem change 
when the average connectivity is increased, (A)— >(F). In fig. 15.21 up. each colored "pixel" 
corresponds to one solution, and each circle to one cluster. As the average connectivity is 
increased, some solutions disappear and the overall structure of clusters changes. This is 
depicted in the six snapshots (A)— ►(F). The magenta clusters are the unfrozen ones, the 
cyan-blue clusters are the frozen ones. Fig. 15.21 down, the corresponding complexity (log- 
number) of clusters of a given entropy, S(s), computed from the 1RSB approach (12.28}) 
for the 6-coloring of random regular graphs. More detailed description of the different 
phases for g-coloring follows. 

(A) A unique cluster exists: For connectivities low enough, all the proper colorings 
are found in a single cluster, where it is easy to "move" from one solution to another. 
Only one possible — and trivial — fixed point of the BP equations exists at this stage 
(as can be proved rigorously in some cases [BG06J). The entropy can be computed 
and reads in the large graph size limit 



(B) Some (irrelevant) clusters appear: As the connectivity is slightly increased, the 
phase space of solutions decomposes into a large (exponential) number of different 
clusters. It is tempting to identify that as the clustering transition. But in this 
phase all but one of these clusters contain relatively very few solutions, as compare 
to the whole set. Thus almost all proper colorings still belong to one single giant 
cluster, and the replica symmetric solution is correct, eq. ( 15. 2ft gives the correct 



(C) The clustered phase: For larger connectivities, the large single cluster decom- 
poses into an exponential number of smaller ones: this now defines the genuine 
clustering threshold q. Beyond this threshold, a local algorithm that tries to move 
in the space of solutions will remain prisoner of a cluster of solutions for a diverging 
time |MS06cj . Interestingly, it can be shown that the total number of solutions is 
still given by eq. (15. 2p . Thus the free energy (entropy) has no singularity at the 
clustering transition, which is therefore not a phase transition in the sense of Ehren- 
fest. Only a diverging length scale (point-to-set correlation length) and time scale 
(the equilibration time) when Cd is approached justify the name "phase transition". 




(5.2) 
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Figure 5.2: Up: Sketch of the structure of solutions in the random coloring problem. The 
depicted phase transitions arrive in the above order on Erdos-Renyi graphs for number 
of colors 4 < q < 8. Down: Complexity (log-number) of clusters of a given entropy, £(s), 
for 6-coloring or random regular graphs. The circles mark the dominating clusters, i.e., 
those which cover almost all solutions. 

(D) The condensed phase: As the connectivity is increased further, another phase 
transition arises at the condensation threshold, c c , where most of the solutions are 
found in a finite number of the largest clusters. Total entropy in the condensed 
phase is strictly smaller than (I5.2p . It has a non-analyticity at c c therefore this is a 
genuine static phase transition. The condensation transition can be observed from 
the two-point correlation functions or from the overlap distribution. 

(E) The rigid phase: As explained in chapter HI two different types of clusters exist. 
In the first type, the unfrozen ones, magenta in fig. 15.21 all variables can take at 
least two different colors. In the second type, frozen clusters, cyan in fig. 15.21 a finite 
fraction of variables is allowed only one color within the cluster and is thus "frozen" 
into this color. In the rigid phase, a random proper coloring belongs almost surely 
to a frozen cluster. Depending on the value of q, this transition may arise before or 
after the condensation transition (see tab. 15.11) . 

(F) The uncolorable phase: Eventually, the connectivity c s is reached beyond which 
no more solutions exist. The ground state energy is zero for c < c s and then grows 
continuously for c > c s . 



78 



CHAPTER 5. COLORING RANDOM GRAPHS 



In table 15.11 we present all the critical values for coloring of Erdos-Renyi graphs, in 
table 15.21 for random regular graphs. Notice the special role of 3-coloring where the 
clustering and condensation transitions coincide and are given by the local stability of 
the replica symmetric solution, see app. O Notice also that for q > 9 in Erdos-Renyi 
graphs and q > 8 in regular graph the rigidity transition arrives before the condensation 
transition. 



q 


c d 


c r 


c c 


c s 


CSP 


Cr(m=l) 


3 


4 


4.66(1) 


4 


4.687(2) 


4.42(1) 


4.911 


4 


8.353(3) 


8.83(2) 


8.46(1) 


8.901(2) 


8.09(1) 


9.267 


5 


12.837(3) 


13.55(2) 


13.23(1) 


13.669(2) 


12.11(2) 


14.036 


6 


17.645(5) 


18.68(2) 


18.44(1) 


18.880(2) 


16.42(2) 


19.112 


7 


22.705(5) 


24.16(2) 


24.01(1) 


24.455(5) 


20.97(2) 


24.435 


8 


27.95(5) 


29.93(3) 


29.90(1) 


30.335(5) 


25.71(2) 


29.960 


9 


33.45(5) 


35.658 


36.08(5) 


36.490(5) 


30.62(2) 


35.658 


10 


39.0(1) 


41.508 


42.50(5) 


42.93(1) 


35.69(3) 


41.508 



Table 5.1: Critical connectivities q (dynamical, clustering), c r (rigidity), c c (condensa- 
tion, Kauzmann) and c s (colorability) for the phase transitions in the coloring problem 
on Erdos-Renyi graphs. The connectivities csp (where the first non trivial solution of 
SP appears) and c r ( m =i) (where hard fields appear at m = 1) are also given. The error 
bars consist of the numerical precision on evaluation of the critical connectivities by the 
population dynamics technique, see appendix [El 
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Table 5.2: The transition thresholds for regular random graphs: Csp is the smallest 
connectivity with a nontrivial solution at m = 0; the clustering threshold q is the smallest 
connectivity with a nontrivial solution at m — 1; the rigidity threshold c r is the smallest 
connectivity at which hard fields are present in the dominant states, the condensation 
c c is the smallest connectivity for which the complexity at m = 1 is negative and c s the 
smallest uncolorable connectivity. Note that 3— coloring of 5— regular graphs is exactly 
critical for that q = 5 + . The rigidity transition may not exist due to the discreteness of 
the connectivities. 

Few more words about the rigidity transition and the rigid phase in coloring. In 
sec. 14.2.21 next to the rigid phase, we also defined the totally rigid phase where almost all 
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the clusters of every size become frozen. And the frozen phase where strictly all clusters 
become frozen. Note that in the random graph coloring the rigidity transition coincides 
with the total rigidity transition for q < 8 for Erdos-Renyi graphs and for q < 7 for regular 
graphs. For larger values of q the rigidity transition is given by the m = 1 computation. 
We have not computed the total rigidity transition for larger q, but it is accessible from 
the present method. The freezing transition is, however, not accessible for the entropic 
1RSB cavity approach. We cannot exclude that in the totally rigid phase there might 
still be some rare unfrozen clusters. 

Note also an interesting feature about the 1RSB entropic solution; in fig. 15.21 down, 
for the connectivity c = 17 the function consists of two branches. The low-entropy 
branch with frozen clusters, and the high-entropy branch with soft clusters. Note that the 
soft branch may also exist for positive values of complexity, e.g. in 4-coloring of Erdos- 
Renyi graphs. We interpreted the gap as the nonexistence of clusters of the corresponding 
size. The gap might, however, be an artifact of the 1RSB approximation which most 
likely does not describe correctly clusters of the corresponding size. For the discussion of 
correctness of the 1RSB solutions see appendix IDl 

0.3 
0.25 

0.2 
0.15 

0.1 
0.05 


12 12.5 13 13.5 14 14.5 

c 

Figure 5.3: Entropies and complexities as a function of the average connectivity for the 
5-coloring of Erdos-Renyi graphs. The replica symmetric entropy is in dashed black, the 
total entropy in red. The complexity of dominant clusters in red. The total complexity, 
computed from the survey propagation, is in dashed blue. 

To make the picture complete we plot the important complexities and entropies as a 
function of the average connectivity, for 5-coloring of Erdos-Renyi graphs see fig. 15.31 We 
plotted in dashed black the replica symmetric entropy f)5.2p . which in coloring is equal 
to the annealed one s ann . The correct total entropy s to t (in red) differs from the replica 
symmetric one in the condensed and uncolorable phase. The complexity of the dominating 
clusters (those covering almost all solutions) Edom (in red, computed at m = 1) is non- 
zero between the clustering and the condensation transition. The total complexity S max 
(in blue), maximum of the curves E(s), can be computed in the region where survey 
propagation gives a nontrivial result. The colorability threshold corresponds to S max = 0. 
We call csp the smallest connectivity at which survey propagation gives a nontrivial result, 
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i.e., the part of the curve E(s) with a zero slope exists. Clusters exists also for c < csp, 
but computing their total complexity is more involved and we have not done it. The 
rigidity transition c r cannot be determined from these quantities. 

In fig. 15.41 we sketch what fraction of solutions is covered by the largest cluster as 
the average connectivity increases for 4-coloring of Erdos-Renyi graphs. In the replica 
symmetric phase c < q the largest cluster covers almost all solutions. In the dynamical 
1RSB phase the largest cluster covers an exponentially small fraction of solutions. In the 
condensed phase the largest state covers fraction of about 1 — m* of solutions^, but this 
part of the curve in not self-averaging. In the uncolorable phase there are no clusters of 
solutions, the ground state is made from one cluster. 



t/: 
O 




Figure 5.4: The fraction of solutions covered by the largest cluster as a function of 
the average connectivity for 4-coloring of Erdos-Renyi graphs. In the condensed phase 
the fraction covered by the largest cluster is not self-averaging and is determined by the 
Poisson-Dirichlet process with parameter m*. 



5.3 Large q limit 

The coloring of random graphs in the limit of large number of colors might seem a 
very unpractical and artificial problem. However, it allows many simplifications in the 
statistical description (rigorous or not) and a lot of insight can be obtained from this 
limit. 

It is known from the cavity method, but also from a rigorous lower [ANP05J and 
upper |Luc91j bound that the colorability threshold for large number of colors scales like 
2glogg. At the same time a very naive algorithm: Pick at random an uncolored vertex 
and assign it at random a color which is not assigned to any of its neighbours, was shown 
to work in polynomial (linear) time up to a connectivity scaling as qlogq. In other words 
this algorithm uses about twice as many colors than needed. Such a performance is not 

1 More precisely from the properties of the Poisson-Dirichlet process, described in sec. 13. 3[ if the 
fraction of solutions covered by the largest state is w then 1 — m* — l/E(l/u?). 
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very surprising, a very naive algorithm performs half as good as possible. The surprise 
comes with the fact that it is an open problem if there is a polynomial algorithm which 
would work at connectivity (1 + e)glogg for an arbitrarily small positive e. 

5.3.1 The 2qlogq regime: colorability and condensation 

The complexity function S(s) at connectivity 

c = 2glogg — logg + 7 (5.3) 
where 7 = 6(1) was computed in [ZDEB-5] and reads 



Us) 



log 2 



log 



elog2 



e{2 + 1 ) + o{e). (5.4) 



where e = l/2q. From this expression it is easy to see that the coloring threshold 
corresponds to 

7s = -1. (5.5) 

and the condensation transition 

7c = -21og2. (5.6) 

Notice, as in [ZDEB-8], that the complexity of the random subcubes model (13.51) . 
sec. 13.11 gives exactly the expression (15 Ah if we take the parameters of the random 
subcubes model as 

p=l-e, a = l + e—^-. (5.7) 

log 2 

This is a striking property of the coloring problem in the limit of large number of colors 
near to the colorability threshold. The 1 — e is a fraction of frozen variables in each 
cluster. Almost all the soft variables can take only one of two colors. The expression 
( 15 Ah means that the soft variables are mutually almost independent and the clusters have 
shape of small hypercubes. And the other way around, this property makes the random 
subcubes model more than just a pedagogical example of the condensation transition. 



5.3.2 The qlogq regime: clustering and rigidity 

Another interesting scaling regime is defined as 

c = g(logg + log log g + a) , (5.8) 

where a = 0(1) is of order one. The large q scaling of the rigidity transition (m = 1) is 
easily expressed from (12. 4ft : 

a r = l. (5.9) 

This was originally computed in [ZDEB-4, ZDEB-5] and |Sem08j . The onset of a non- 
trivial solution for the survey propagation corresponds to the rigidity transition at m = 
and reads |KPW04j 

a SP = l-log2. (5.10) 

2 We remind that in the section f3. II entropies were logarithms of base 2 whereas everywhere else they 
are natural logarithms. 
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Figure 5.5: We plotted the difference (c r — Cd)/q = ct r — «d and (c r — csp)/q = a r — asp. 
The data are takes from table 15.11 The difference a r — «sp indeed seems to converge to 
the theoretical log 2, the difference a r — «sp seems to converge to around 1/4. 

An empirical observation is that for q = 3 the threshold for survey propagation is smaller 
than the rigidity at m = 1, but for q > 4 the order changes and the distances between the 
two threshold grows with q. Based on this observation we conjectured that the clustering 
transition is 

l-log2 > a d > 1. (5.11) 

Note that recently the dynamical transition was proved to be 1 - log 2 > a d |Sly08| . 
Fig. 15.51 actually suggest that ~ 1/4. Its precise location is actually an interesting 
problem because it could shed light on the way soft fields converge to hard fields in the 
cavity approach. 

Concerning the total rigidity transition, where almost all the clusters of all sizes 
become frozen, we have not manage to compute it in the large q limit. It is not even clear 
if the relevant scaling is as ( 15. 81) . The same is true for the even more interesting freezing 
transition, where all the clusters become frozen. 

5.4 Finite temperature 

It is interesting to study how does the antiferromagnetic Potts model, coloring at zero 
temperature, behave at finite temperature. In particular which of the zero temperature 
phase transitions survive to positive temperatures and what do they correspond to in the 
phenomenology of glasses. This has been done in [ZDEB-6] and we summarize the main 
results here. 

The belief propagation equation for coloring (12.51) generalizes at finite temperature to 
= ^=J II [M 1 - ^) = ^M^}) ■ (5-12) 

kEdi—j 



The distributional 1RSB equation ( 12.241) is the same. 
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• The clustering transition — becomes the dynamical phase transition Td at pos- 
itive temperature. The notion of reconstruction on trees, introduced in sec. 12.1.11 
generalizes to positive temperatures. Constraints then play the role of noisy chan- 
nels in the broadcasting. The dynamical temperature Td is then defined via di- 
vergence of the point-to-set correlations (I2.22p . Or equivalently via the onset of 
a nontrivial solution of the 1RSB equations at m = 1. At the dynamical transi- 
tion the point-to-set correlation length and the equilibration time diverge. There 
is however no non-analyticity in the free energy, Ehrenfest might thus not call it a 
phase transition. 

• The condensation transition — becomes the Kauzmann phase transition Tk 
at positive temperature. The point at which the complexity function at m — 1 
(structural entropy) becomes negative defines the Kauzmann temperature |Kau48j . 
At the Kauzmann temperature the free energy has a discontinuity in the second 
derivative. This corresponds to the discontinuity in the specific heat. Kauzmann 
transition is thus genuine even in the sense of Ehrenfest. 

• The rigidity transition — is a purely zero temperature phase transition. At 
positive temperature the fields (15.121) cannot be hard. 

• The colorability transition — is a purely zero temperature phase transition. 
At the colorability threshold the ground state energy becomes positive (it has dis- 
continuity in the first derivative). At a finite temperature, however, there is no 
corresponding non-analyticity. 




Figure 5.6: Phase diagrams for the 3-state (left) and 4-state (right) anti-ferromagnetic 
Potts glass on Erdos-Renyi graphs of average degree c (top) and regular graphs of degree 
c (bottom). For q = 3 the transition is continuous T d = T K = Xi ocal . For q = 4, we find 
that Td > Tk > T\ oca \, while for larger connectivities these three critical temperatures 
become almost equal. The Gardner temperature Tq for regular graphs is also shown 
(green), bellow Tq the 1RSB solution is not correct anymore (for Erdos-Renyi graph we 
expect this curve to look similar). The bold (red) lines at zero temperature represent the 
uncolorable connectivities c> c s . 

Fig. 15.61 shows the temperature phase diagram of 3- (left) and 4-coloring (right) on 
both Erdos-Renyi (up) and regular (down) random graphs. The dynamical temperature 
is in blue, the Kauzmann temperature in black. 
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The temperature at which the replica symmetric solution becomes locally unstable, 
see appendix EJ is called Ti oca i. In the terms of reconstruction on trees this is the Kesten- 
Stigum bound [KS66a, KS66bJ. This temperature is a lower bound on the dynamical 
temperature T^, but also on the Kauzmann temperature Tr. This is because bellow 
^locai the two-point correlations do not decay, which is possible only bellow T#. Note 
that in the 3-coloring Td = Tk = Tiocai and this phase transition is continuous in the 
order parameter P l_> J \ip % ^i) ( 12.24ft . For q > 4 colors we find instead Td > Tk > Ti oca i 
and the dynamical transition is discontinuous. At large connectivity, however, the three 
temperatures are very close, see fig. 15.61 where the Ti ocal is in pink. 

Correctness of the 1RSB solution — The last question concerns correctness of 
the 1RSB solutions itself. The local stabilities of the 1RSB solution are discussed in 
appendix [D] The temperature at which the 1RSB solutions becomes type II locally 
unstable, see appendix O, is called the Gardner temperature Tq |Gar85] . We computed 
it only on the ensemble of random regular graphs, see fig. 15.61 the Tq is in green. We do 
not know how to compute the stability of the type I, but we argued that the corresponding 
critical temperature should be smaller than the Ti oca i- An important consequence is that 
in the colorable region the 1RSB solution is stable for q > 4 coloring. 

Coloring with three colors is a bit special, as Ti oca i = Td = Tk- However, at small 
temperatures the stability of type I can be investigated from the energetic approach, again 
discussed in app. [Dl It follows that at least in interval c G (c s , cq) = (4.69, 5.08) the 1RSB 
solution is stable at low temperature. For c > cq on contrary the Gardner temperature 
is strictly positive. We cannot exclude that part of the colorable phase is unstable, but 
in such a case the unstable region would have a sort of re-entrant behaviour. Moreover 
the ferromagnetic fully connected 3-state Potts model has also a continuous dynamical 
transition Td = Ti oca \ yet it is 1RSB stable near to Td [GKS85J . We thus find more likely 
that also the colorable phase of 3-coloring is 1RSB stable. 

Finally, the local stability is only a necessary condition. The full correctness of the 
1RSB approach have to be investigated from the 2RSB approach. We implemented the 
2RSB on the regular coloring, the results are not conclusive, as the numerics is involved, 
but we have not found any sign for a nontrivial 2RSB solution in the colorable region. 



Conclusions and perspectives 



In this final section we highlight the, in our view, most interesting results of this thesis. 
More complete overview of the original contributions is presented in sec. M.iA Scientific 
research is such that every answered question raises a number of new questions to be 
answered. We thus bring up a list of open problems which we find particularly intrinsic. 
Finally we give a brief personal view on the perspective applications of the results obtained 
in this work. 



Key results 

The main question underlying this study is: How to recognize if an NP-complete problem 
is typically hard and what are the main reasons for this? 

In order to approach the answer we studied the structure of solutions in random 
constraint satisfaction problem - mainly in the graph coloring. We did not neglect the 
entropic contributions, as was common in previous studies, and this led to much more 
complete description of the phase diagram and associated phase transitions, see summa- 
rizing fig. 15.21 

The most interesting concept in these new findings was the freezing of variables. We 
pursued its study and investigated its relation to the average computational hardness. 
We introduced the locked constraint satisfaction, where the statistical description is easily 
solvable and the clustered phase is automatically frozen. We indeed observed empirically 
that these problems are much harder than the canonical K-satisfiability. They should 
thus become a new challenge for algorithmical development. As we mention in the per- 
spectives, we also anticipate that the locked constraint satisfaction problems are of a 
more general interest. 



Some open problems 

(A) Clusters and their counting on trees — In sec. |2]we derived the 1RSB equa- 
tions on purely tree graphs. Our derivation was, however, not complete as it is not 
straightforward why the complexity function should be counting the clusters as we de- 
fined them on trees. More physically founded derivations are for example the original 
one [MPOOj . And also the one presented in |MM08j where the complexity is shown to 
count the fixed points of the belief propagation. We are, however, persuaded that the 
purely tree approach is more appealing from the probabilistic point of view, as treating 
correlations in the boundary conditions on trees is easier than treating the random graphs 
directly, for a recent progress see e.g. |Sly08 IGM071 IDM08J. This is why we chose to 



present this derivation despite its incompleteness. 
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CONCLUSIONS AND PERSPECTIVES 



In general we should say that creating better mathematical grounds for the replica 
symmetry breaking approach is a very important and challenging task. 

(B) What is the meaning of the gap in the E(s) function — We computed 
the number of clusters of a given entropy via the 1RSB method. For some intervals 
of parameters there is no solution corresponding to certain intermediate sizes. In other 
words there is a gap in the 1RSB function E(s). See e.g. fig. 15.21 we observed such a gap 
in many other cases. Does this gap mean that there are truly no clusters of corresponding 
sizes or does it mean that the 1RSB method is wrong in that region or is there another 
explanation? 

(C) Analysis of dynamical processes — In this thesis we described in quite a detail 
the static (equilibrium) properties of the constraint satisfaction problems. Very little is 
known about the dynamical properties - here we mean both the physical dynamics (with 
detailed balance) and the dynamics of algorithms. Focusing on results described here: the 
dynamics of the random subcubes model can be solved [ZDEB-8], and the uniform belief 
propagation decimation can be analyzed [MRTS07J, see also appendix IF. 1.21 However in 
general even the performance of simulated annealing as a solver is not known. And the 
understanding of why the survey propagation decimation works so well in 3-SAT and not 
that well in other problems, e.g. the locked problems or for larger K, is also very pure. 

The most exciting conjecture of this work is the connection between the algorithmical 
hardness and freezing of variables. Several indirect arguments and empirical results were 
explained in sec. 14.41 to support this conjecture. It is, however, not very clear what is the 
detailed origin of the connection between presence of frozen variables in solutions and the 
fact that dynamics (of a solver) does not seem to be able to find them. 

(D) Beyond random graphs and the thermodynamical limit — For practical 
application the perhaps most important point is to understand what is the relevance 
of our results for instances which are not random or not infinite. For example fig. 12.21 
suggests that even on small random instances the clustering can be observed and is thus 
probably relevant. We also observed that the solutions-related quantities seems to have 
stronger finite size effects than the clusters-related properties, compare e.g. fig. 11.31 with 
fig. 14.11 This is an interesting point and it should be pursued. 

Perspectives 

This work should have a practical impact on the design of new solvers of constraint 
satisfaction problems. Instances with only frozen solutions should be used as new bench- 
marks for SAT solvers. At the same time where the design allows such instances should 
be avoided. 

More concretely, the belief propagation algorithm is used as a standard approximative 
inference technique in artificial intelligence and information theory. One of the important 
problems with applications of the belief propagation is the fact that in many cases it does 
not converge. Many converging modifications were introduced. In might be interesting 
to investigate in this context the reinforced belief propagation, see appendix IF.2.31 which 
sometimes converges towards a fixed point when the standard belief propagation does not. 
As the reinforcement algorithm seems to be very efficient, robust and is not theoretically 
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well understood different variants of the implementation should be studied empirically. 
It would be interesting to see if this algorithm performs well on non-random graphs, 
or if it can provide information useful for the practical solvers. Several other concepts 
enhanced in this thesis might show up useful in algorithmic applications. We feel that 
the whitening of solutions might be one of them. 

We introduced the locked constraint satisfaction problems as a new algorithmical 
challenge. Moreover the simplicity of their statistical description makes accessible several 
quantities which are difficult to compute in the if- SAT problem. For example the weight 
enumerator function or the x-satisfiability threshold. But these new models are exciting 
from many other points of view. Their hardness might be appealing for noise tolerant 
cryptographic applications. Planted ensemble of the locked problems might be a very 
good one-way functions. The fact that the solutions of the locked problems are well 
separated makes them excellent candidates for nonlinear error correcting codes. It will 
be interesting to investigate if they can be advantageous over the standard linear low- 
density-parity-check codes [Gal62l lGa!68l IMN951 iMonOT] . 

Clusters of solutions come up naturally in the pattern recognition and machine learn- 
ing problems. There each cluster corresponds to a pattern which should be learned or 
recognized. Similarly the different phenotypes of a cell might be viewed as clusters of 
fixed points of the corresponding gene regulation network. The methods developed in 
this thesis might thus have impact also in these exciting fields. 
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Appendix A 

1RSB cavity equations at m = 1 



Here we derive how the 1RSB equation (12.241) simplifies at m = 1 for the problems 
where the replica symmetric solution is not factorized. We restrict to the occupation 
models, but a generalization to other models is straightforward. Advantage of these 
equations is that the unknown object is not a functional of functionals but only a single 
functional. Moreover, the final self-consistent equation does not contain the reweighting 
term. This simplification makes implementation of the population dynamics at m — 1 
much simpler, and thus the computation of the clustering and condensation transitions 
easier. Derivation of the corresponding equations for the i^-SAT problem can be found 
in [MRTS08j . 

Write the RS equation ( 11. 17ft for the occupation problems in the form 



{.Sj} j&da—i \bedj~a / 



where the constraints C a ({sj}, Sj) = 1 if 5^ ■ Sj + Sj G A, and otherwise. Let Prs('0) be 
the distribution of RS fields over the graph. 
The 1RSB equations (12. 24ft at m = 1 are 



= ^-j n n w^p^^ h 



j&da—i bddj—a 

Z^\{^}) 5 [r^ 1 - Hi^})} = ^2({P b ^}) ■ (A.2) 



The averages over states 

= / dvc^xvc 4 ) €r (a.3) 



satisfy the RS equation (1A.1I) . And consequently the RS and 1RSB normalizations are 
equal = Z^ % . The full order parameter is the probability distribution of P's over 
the graph, it follow the self-consistent equation 



, K-l U 

II II {d^ W ^rsbW)]} W) - • (A.4) 



i=l i=l 
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We define the average distribution P(ip\ip) on those edges where the RS field is equal 
to a given value ip 



P(4)\iP)VbsM = / dP(^)P 1RSB [P^)}P^)5 



ip- dipP(i/))ip 



(A.5) 



Now we rewrite all the terms on the right hand side using the incoming fields and distri- 
butions, i.e., using first eq. (IA.4I) and then (IA.2I) . 



p(V#)p rs (V) 



{1} J i=l i=i 

F 2 ({P j })S J d^-P 2 ({P J })^ 

K-l L 



e?(w) / nn{ dpj (^>iRSB[^(^)] 

{1} J i=l 3=1 

K—l I- 



i=l j=l 



K-l k 



e?(w) / nnK PRs( ^ 

{1} J i=l i=i 

T3\ 



i=i i=i 



(A.6) 



where the original Dirac function was rewritten using 



. K-l i 



i=l j=l 
.K-l k 



\ I n n wp j w\ hw) = nw)) , (A.?) 

J i=l j=l 



and in last equality was obtained using the integral of eq. f]A.5|) 



dV>P(V#)PRs(VO = / dP(^)P 1RSB [P(^)]P(^) 



(A.8) 



To simplify the equations further, in particular to get rid of the reweighting term 
Z({ip J }), we define a distribution P s 



v> s p.(v#) = i> s p(i>m => p(v#) = E t.p.Mty , (a.9) 
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then by factorizing the sum over components s we get 

{/} J i=l j=l 

Ec(M,») U ^Bf" 

K-l h 

n n d^c^i?)] ^ - • (a.io) 
i=i j=i 

This final equation might look more complicated than the original one, but, in fact, it is 
much easier to solve. It could seem that we need a population of populations to represent 
the distribution P s (iP\iI))Vb,s{' 1 P) ■ But keeping in mind that the proper initial conditions 
are 



Wo 



(All) 



independently of the RS field ip we see that the probability distribution P s (V'I'0)'Prs('0) 
may be represented by a population of triplets of fields - the first one corresponding to 
the RS field if) and the other two corresponding to the two components ( 1A.11I) . 

In the population dynamics we first equilibrate the RS distribution "Prs(^) and then 
initialize the other two components according to flA.111) . In every step of the update we 
first fix randomly the set of indexes {j} and compute the new ip, then given the value 
s we choose the set of indexes {s^ according to a probability law given by the first line 
of eq. ( 1A.10I) . then we compute the new ip for s = and s = 1 and change a random 
triplet in the population for the new values. In summary, eq. (lA.lOj) allows to reduce the 
double-functional equations at m = 1 into a simple-functional form, which is much easier 
to solve. 

The internal entropy s = srs — S, and thus also the complexity function, may be 
computed by making very similar manipulations as 



k i 



{/} J i=lj=l 

K k 

nn [d^>i(^'i?)] \o g z a+9a ({^}) 

i=l j=l 



,M E {Si} C({s t })llliIlU< 



Z a + da ({ip J }) 



Es(oc-i) /n[d?^ RS (?) 

I J i=l 



i=i 



(A.12) 



We can also express other quantities, e.g. the inter q = qns and intra qi state 
overlaps. 

qi = J dP(^)P 1RSB j d«^i = ^ J dtfVns^sJ dV>P s (V#)^- (A.13) 
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Factorized RS solution — Several times, see e.g. sec. 14.3.31 we used the equations 
at m = 1 for problems with factorized RS solution, "Prs(V0 = ^C0 — ^0- The derivation 
is straightforward from flA.lOp 



p.sW = e q(u})^y e s ) n n < / n n ^ - w)) • 

« ^ s {s,} t=l j=l 7 i=l 3=1 



_(A.14) 

Proper initial conditions for the population dynamics resolution of (1A.14I) is P s {ip s = 
1) = 1. 

At zero temperature the distributions can be written as the sum of the frozen and 
soft part 

P x ^) = frSty - Q) + (1 - /xi)A(V0 , (A.15a) 
PoO) = - LJ ) + (1 - ^o)Po(^) • (A.15b) 

Self-consistent equations for the fractions of hard fields /ii, /io (I4.20atl4.20bp follow from 

(jm. 



Appendix B 

Exact entropy for the balanced LOPs 



Rigorous results about the entropy and the satisfiability threshold can be obtain com- 
paring the first and second moment of the number of solutions, that is: If a number of 
solution on a graph G is Mg then the first moment is average over the graph ensemble: 



(jV G ) = Prob ({a} is SAT) . 
W 



The second moment is 



{Mg)= Yl Prob ({ax} and {a 2 } are both SAT) . 

Wl},W'2} 



(B.l) 



(B.2) 



The Markov inequality then gives an upper bound on the entropy and the satisfiability 
threshold 

Prob(A/G > 0) < (Af G ) ■ (B.3) 
The Chebyshev's inequality gives a lower bound via 



Prob(A/" G > 0) > 



(B.4) 



B.l The 1 st moment for occupation models 



Let us remind that the occupation models are defined via a (K + l)-component vector 
A, such that = 1 if and only if there can be % occupied particles around a constraint 
of K variables. We consider by default A = A K = 0, i.e., that everybody full of empty 
is not a solution. We also consider all the M constraints are the same. We have Q(l)N 
variables of connectivity /, where J2iloQ(l) = 1 an d I = Y^LqIQQ) = KM/N. 

In order to compute the first moment we divide variables into groups according to their 
connectivity and in each groups we choose fraction ti of occupied variables. Number of 
ways in which this is possible is then multiplied by a probability that such a configuration 
satisfies simultaneously all the constraints. 

JV£,f(i-*i)Q(0 

n)...(K - r M ) 



<«=r« S n(,S), i £i.,..-. 1 (,; 

5[J2 r a-~ ltN ) s[tlN-J2 lt iQ( l )N) ,(B.5) 



IN 
K...K 



,B=1 
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APPENDIX B. EXACT ENTROPY FOR THE BALANCED LOPS 



where t is the total fraction of occupied variables, this variable might seem ambiguous, 
as it can be integrated out, but we will appreciate its usefulness later, r a is a number of 
occupied variables in a constraint a. 

We develop expression ( 1B.5I) in the exponential order. In order to do so we exchange 
the last two delta functions by their Fourier transforms, introducing two complex La- 
grange parameters logx and logu. 

(Af G ) « J dt JH dt ij dx J dnexpivj -^Q(0falogt z + (l-t0log(l-*0] 



+ Z [tlogt + log (l-t)]+ logu 



J2^iQ(l)-fl 



I 



r=l ^ ' 



X 



- tl log X 

Saddle point with respect to parameters ti gives us 



(B.6) 



ti 



u 



l + t l 



(B.7) 



and we call p A {x) = Y^r=x &{A r — 1) (5)^- Using this we have 



<Xt (\.r <\u cx'p.V^ -^-\ogp A {x) —tllogx 



+ ^Q(/)log(l + u / )-tIlogM + I[tlogt+(l-t)log(l-t)] }. (B.8) 

The saddle point equations read 

d u : t ~- 



d x : t 
d t : t 



xd x p A (x) 

Kp A (x) 
xu 

1 + xu ' 



(B.9a) 

(B.9b) 
(B.9c) 



As the parameter t is the only physically meaningful from the three, the goal is to 
express the annealed entropy as a function of t and find its maxima. We do that by 
inverting numerically flB.9al) and plugging (IB.9cl) in (IB. 81) . Eq. flB.9cp then express the 
saddle point with respect to the parameter t. We can write 



SannW = Yl ^(0 ^g [1 + u(t) 1 ] + — \ogp A (t) 



where 



p A (t) = j^5{A r -l)(^ ( 



u(t) 



(1-*) 



K-r 



(B.10) 



(B.11) 



B.2. THE 2 nd MOMENT FOR OCCUPATION MODELS 
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where u(t) is an inverse of (]B.9al) . 

For the regular graphs Q(l) = 5(1 — L) the inverse of (!B.9aP is explicit u = [t/(l — t)] 1 / 1 
and thus 

K 



"5 ann reg 



W = ^ 1 °g|E^- 1 )(f) [t r (l-t) K - r ]^\. (B.12) 



B.2 The 2 nd moment for occupation models 

The second moment is computed in a similar manner. First we fix that in a fraction t Xy i 
of nodes of connectivity I the variable is occupied in both the solutions 01, a 2 in (IB. 21) . In 
a fraction t y j the variable is occupied in o\ and empty in cr 2 and the other way round for 
t z ,i- We sum over all possible combinations of < t Xt i,t y j,t Z) i such that ^2 w=xy z t w ,i < 1- 
All this is multiplied by the probability that such two configurations 01,02 both satisfy 
all the constraints. 

(A® = J dt x dt y dt z { ^ w E^ } II ( {tx MDN) (£q("n) (t z mw) 

M 

E E E II 5 (^,a+r„, a - mAr x , a+ r z , a - 1) 

^x,i'---' r ^,M r Vl i,...,r ViM r z ,!,...,r ZyM a=l 



( K ~ E W=X ,y, Z r W ,l) ...{K- E W=X ,y, Z r wM)J w = xy , Z V r --l • • • 

( K JN K ) II ^fc^" 1 ^) ^iJ^-E^^W^ .(B.13) 

' w=x,y,z \a=l / \ I / 

We introduce Fourier transforms at a place of both the Dirac functions, the conjugated 
parameters are log x, log y, log z for the first Dirac function, and logw^, logu^, logw z for 
the second one. After that we suppress the parameters t W} i in the same manner as we did 
for the first moment. We obtain for the second moment entropy 

S 2 nd = l[t X lOgt x +t v lOgty+t z \Ogt z +(l-t x -ty-t Z )\Og(l-t x -ty-t z )] 

-l(t x \ogx + t y \ogy + t z \og z) + ^- log p A (x,y,z) 

+ Q( l ) lQ g i 1 + u x + u l + u i) ~ ~ l (t* ^gu x + t y log u y + t z \ogu z ) , (B.14) 
i 

where 

K min(ri,r 2 ) , v 

p A (x,y,z)= s(A ri A r2 -i) Yl ( iri _ s)ir2 _ s)s ) xS y iri ~ s)z(r2 ~ s) 

ri,r2=0 s=max (0,ri+r2--ft") ' 



(B.15) 



98 



APPENDIX B. EXACT ENTROPY FOR THE BALANCED LOPS 



and the saddle point with respect to t w , w and u w [w = x, y, z) is 



dt w ■ t w 
d,. : wu„, 



+ U l x + U l y +U l z 



w = x,y,z, 



wd w p A (x,y,z) 
Kp A (x,y,z) 

1 ty t Z 



w = x,y,z, 



, w = x,y,z. 



(B.16a) 

(B.16b) 
(B.16c) 



Once again the parameters t w are physically meaningful, so we want to express S2nd 
as a function of these. We thus need to inverse ( 1B.16al) . note that such an inverse is well 
defined, and using flB.16cp we obtain 



S2nd(tx,t y ,t z ) = -^\ogp A (t x ,t y ,t z ) + ^2Q(l) log I 1 + ^2 [ u w(t x ,t y ,t z 

wd{x,y,z} 



(B.17) 



i 



where 

K min(ri,r2) 

PA(t x ,t y ,t z ) = ^ S(A ri A r2 -l) ^ 



ri,r 2 =0 
(ri-s) 



t. 



=max (0,ri+r2— K) 
(r-2-s) 



K 



(ri - s)(r 2 - s)sj \u x (t x ,ty,t z ) 

(B.18) 



'1 _ + _ + _ + \(K-ri-r 2 +s) 
„ ' •'I •'j/ L z) 



The global maximum with respect to t x ,t y ,t z needs to be found. 

For the regular ensemble Q(l) = 5(1 — L) the function flB.16aj) is explicitly reversible 
and the final expression for the second moment entropy simplifies significantly 



^2nd,reg \txi tyi tz 



L 
K 



log E 



K'.S(A ri - 1)S(A,. 2 - 1; 



(ri — s)\ (r 2 — s)\ s\ (K — r\ — r 2 + s)\ 



where the range of summations is the same as in (IB. 18[) . 



L-l 
L 



(B.19) 



B.3 The results 

The main result is that for some of the symmetric (Ax- r — A r for all r = 0, . . . , K) and 
locked occupation problems (Q(0) = Q(l) = 0) the first and second moments compu- 
tation leads the exact entropy of solutions f)4.18p . And thus also the exact satisfiability 
threshold. The cases where this statement holds are marked by a * in tab. 14.11 and we 
call them balanced LOPs. We observed that some of the balanced problems A are created 
iteratively starting from 010 or 01010 and adding 

A K+2 = 0A K , A K+4 = OlA^lO . (B.20) 

We, however, found also other balanced cases than flB.20h . The simplest example of 
symmetric locked problem which is not balanced is A = 010010, and many others of 
higher K. 

Let us now show this result. For all the symmetric occupation problems: 



B.3. 
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• The annealed entropy (IB. 101) has a stationary point a.t t = 1/2 (u = 1, x = 1). At 
this stationary the entropy evaluates to (14.181) . 

• The second moments entropy ( IB. 171) has a stationary point at t x — t y — t z — 1/4 
(u x — u y — u z — 1, x — y — z — 1). At this stationary point the second moment 
entropy evaluates to twice the (14.181) . To prove this statement observe that for 
the symmetric problems p^(l/4, 1/4, 1/4) = [p^(l/2)] 2 . This last identity can be 
derived from the Vandermonde's combinatorial identity 

©-fi(X:?)- 

• The second moment entropy has another stationary point at t x = 1/2, t y = t z = 
or t x — 0, t y — t z — 1/2. This stationary point is equal to the first moment entropy 
at t = 1/2. 

In the problems where one of the above stationary points is the global maximum the 
annealed entropy is exact and the satisfiability threshold easily calculable from (14.181) . 

In the symmetric problems with leaves (Q(l) > 0), or those which are not locked (e.g. 
0110) or not balanced (e.g. 010010) another competing maximum of the second moment 
entropy appears before the annealed entropy goes to zero. 

We investigated numerically that this does not happen for the balanced problems 
described by the recursion (1B.20|) . So far we were not able to prove this last point 
analytically. This is, however, a technical problem, much simpler that the original one. 

The main message of this analysis is what are the ingredients of the model which 
make the satisfiability threshold accessible to the second moment computations. Here we 
showed that it is on one hand the (unbroken) symmetry of the problem and on the other 
hand the point-like clusters. Such a general result might be surprising because otherwise 
the satisfiability threshold is known exactly in only a handful of the NP-complete problems 
[ACIMOll lMZK+99al IAKKK011 ICM04] . 
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Appendix C 

Stability of the RS solution 



In chapter [2] we argued in detail that the replica symmetric solution is correct if and only 
if the point-to-set correlations decay to zero, or equivalently if the reconstruction is not 
possible. Failure of the RS solution may (but does not have to) manifest itself via the 
divergence of the spin glass susceptibility. In a system with Ising variables G { — 1, +1} 
this is defined as 

hi 

where (-) c is the connected expectation with respect to the Boltzmann measure. 

Originally the replica symmetric instability was investigated from the spectrum of the 
Hessian matrix in a celebrated paper by de Almeida and Thouless |dAT78j . Equivalence 
between the RS stability and the convergence of the belief propagation equations on a 
single large graph is also often stated. In the reconstruction on tress this corresponds to 
the Kesten-Stigum condition |KS66al KS66bJ. It is not straightforward to see that all 
these statements are equivalent. We thus try to put a bit of order to the different ways 
of expressing the stability of the RS solution.]. 



C.l Several equivalent methods for RS stability 



Susceptibility chains — Perhaps the most direct way how to investigate the diver- 
gence of the spin glass susceptibility ( 1C.1I) is to write 



Xsg « ^E(( SlSo )D « ^ 7 d E(( S , So ) 



2 ) 

cl i 



(C.2) 



where Sq is a typical variable (the origin), Sd is a variable at distance d from sq, and 7 d 
is the typical number of variables at distance d from s (7 = I 2 /I — 1). The average E(-) 
is over the randomness of the graph. The spin glass susceptibility diverges if and only if 
A > 1 where 



A = 7 lim E({s d s ) 2 c ) 

a— >oo L 

Using the fluctuation dissipation theorem we can rewrite 



(C.3) 



dh 
dh d 



E 



n 

i=i 



dhj-i 
dhi 



(C.4) 



1 This overview has been worked out in collaboration with F. Krzakala and F. Ricci-Tersenghi. 
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where h$, . . . , hd is a sequence of cavity fields (11.341) on the shortest path from s to Sd- 
The dependence of the cavity field hi on hi-\ is given by the belief propagation equations. 
This method to investigate the RS stability was used e.g. in [MMR05J or [ZDEB-1]. It is 
numerically involved and not very precise as in practice d can be taken only at maximum 
10-20. 



Noise propagation — Call the contribution to the spin glass susceptibility from 
the layer of variables at a distance d from 

k,\k,0\=d i£dO fc,|Jfc,t|=d-l iedo 

where hk are cavity fields at distance d from h , and the sum is over all the cavity fields 
needed to compute ho- The spin glass susceptibility diverges if and only if the numbers 
Vd are on average growing with the distance d. 

The evolution of numbers v can be followed via the population dynamics method. 
Next to the population of fields h we keep also a population of positive numbers v. When 
a field h is updated according to the belief propagation equations, we update also the 
number v according to (1C.5I) . The RS solution is stable if and only if the overall sum £\ v % 
is decreasing during the population dynamics updates. This method was implemented 
e.g. in |MS06aj or [ZDEB-3]. It is simple and numerically very precise. 



Deviation of two replicas — Consider a general form of the belief propagation equa- 
tions h = f({hi}). After averaging over the graph ensemble we obtain distributional 
equations fll.23aHl.23bp which are solved via the population dynamics technique. Con- 



sider now two replicas of the resulting population, each element i differs by Shi. Keep 
running the population dynamics on both these replicas and record how the differences 
Shi are changing 

The differences Sh can be negative and positive. Take v = (Sh) 2 then 



v 



The second term can be neglected because the terms Shi and Shj are independent. This 
brings us back to the equation (1C.5I) . 

Thus the replica symmetric solutions is stable if and only if the two infinitesimally 
different replicas do not deviate one from another. This method is very fast to implement 
and is thus useful for preliminary checks of the RS stability. 



Convergence of the belief propagation — The stability of replica symmetric so- 
lutions is equivalent to the convergence of the belief propagation equations on a large 
random graph. This fact follows directly from the previous paragraph. Eq. fl0.6j) gives 
the rate of convergence (divergence) of two nearby trajectories of the dynamical map 
defined by the BP iterative equations. 
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Variance propagation — Often a "variance" formulation of the stability if described. 
Assume that instead of a value hi on every link, there is a narrow distribution of values 
g(hi) parameterized by a mean hi and a small variance How does h and v evolve? We 
have now 

h = Jdhg(h)h = JUidhigiihi)}/^}), (C.8) 
v = J dhg{h){h-hf = y n [dWW'KM) " (E) 2 , (G.9) 

where h = f({hi}) is the belief propagation equation. However, since the variance is 
infinitesimal, the variation of hi around hi is very small, so that 

fdhi}) = /({£» +E ft - ^ir\' ( c - 10 ) 

i 

and therefore one obtains h = f({hi}) and 

-i>(m) 2 ' 

which is nothing else then equations (|C.5p . 

Numerical instability towards the 1RSB solution — The RS stability can also 
be investigated from the numerical stability of the trivial solution of the 1RSB equations. 
Indeed if the distribution of fields over states is regarded the probability distribution of 
a small variance g(h) then the 1RSB equation (12.241) gives for a p th moment of g(h) 

W= ^zj \\[^ i g l {h l )}Z m {{h l })f%{h l }) , (C.12) 

where Z is the normalization of the BP equations and its m th power is the reweighting 
factor. Expansion gives 

Z m ({hi}) = Z m {{hi}) + mZ m -\{hi}) T - hi) dZ ^ l}) (C.13) 



hi 



The equations for the variances (IC.lip does not depend on the second term from (10. 13[) . 



as this is of a smaller order. As a consequence the condition for stability is independent 
of the parameter m. 

It is quite remarkable fact that the divergence of the spin glass susceptibility corre- 
sponds to the appearance of a nontrivial solution of the 1RSB equation at all the values 
of m. In particular because we observed that when the instability is not present the onset 
of a nontrivial 1RSB solution is m dependent, see e.g. fig. ID. 21 

The eigenvalues of the Hessian — The replica symmetric solution is a minimum of 
the Gibbs free energy. This is often investigated from the spectra of the matrix of second 
derivatives called Hessian. The equivalence between this approach and the divergence of 
the spin glass susceptibility is a classical result, see e.g. the book of Fischer and Hertz 
[FH9T] . page 98-100. 
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C.2 Stability of the warning propagation 

At zero temperature the necessary (but not sufficient) condition for the replica symmetric 
solution to be stable is the convergence of the warning propagation on a single graph. 
Obviously if the warning propagation does not converge then BP does not either, and 
convergence of the BP is equivalent to the replica symmetric stability Advantage of the 
investigation of the warning propagation convergence is that it can be treated analytically, 
without using the population dynamics method. 

Consider a model with Ising spins where the warnings u (ll.35bj) can take only three 



possible values u G {— 1,0,1}. Consider warning u and one of the warnings on which 
u depends, say u$. Except m the warning u depends also on u\,...,Uk, where k is 
distributed according to Q{k). The degree distribution conditioned on the presence of 
two edges is 

e(*-2) = fc^Q(*). (C.14) 
k — k 

Call P(a — > b\c — + d) the probability that the warning u changes from value a to value b 
provided that the warning uq was changed from value c to value d. This probability can 
be always computed from the probabilities p_,p ,p + that a warning u = —1, 0, +1 

P ( a ^ h \ c ^ d ) = J2Q(k)P k (p_,p ,p + ;a->b\c^d), (C.15) 

k 

where the function P k depends on the model in consideration. This probability describes 
a proliferation of a "bug" in the warning propagation. We define a bug proliferation 
matrix Py of dimension 6, i = a — > b, j = c — > d. The stability of the warning 
propagation is then governed by the largest (in absolute value) eigenvalue of this matrix 
Amax- The warning propagation is stable if and only if 

7Amax<l, (C.16) 

where 7 = k 2 /k — 1 is the growth rate of the tree (7 is the typical number of vertices 
at distance d from the root). This analysis is often called bug proliferation [KPW041 
MMZ06J (mostly in the context of the 1RSB stability). This investigation of the warning 
propagation stability was used e.g. in [ZDEB-1] or [CKRT05J. 

An example where the warning propagation is stable, however, the belief propagation 
is not, can be found in [ZDEB-3] for the 1-in-K SAT problem. In 1-in-K SAT the warning 
propagation stability threshold corresponds to the unit clause propagation upper bound 
[ZDEB-3]. 
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Concerning the correctness of the 1RSB solution: the Boltzmann measure is split into 
clusters. This leads to an exact description of the system if and only if both the following 
conditions are satisfied. 

• Condition of type I — the point-to-set correlation with respect to the measure over 
clusters decay to zero. The statistics over clusters may be described on the replica 
symmetric (tree) level. Clusters do not tend to aggregate. 

• Condition of type II — the point-to-set correlations within the dominating clus- 
ters decay to zero. The interior of these clusters may be described on the replica 
symmetric (tree) level. Clusters do not tend to fragment into smaller ones. 

Within the cavity approach these conditions can be checked from the 2RSB equation 



where the functional Ti is given by the 1RSB equation (12.241) . We call the solution of 

flTXll trivial if either P^ j [P l ^i] = 5[P l ^ j ] or each P^'^^O = Sty"' - ^ J ), where 
the P l ~^i is the solution of f)2.24p . If and only if the (population dynamics) solution of 
the 2RSB equation at m = m*, m 2 = 1 and at m — 1, m 2 = m* is trivial then the two 
conditions are satisfied, and the 1RSB solution at m* is correct. 

Solving the 2RSB equation is, however, numerically involved. Even on random regular 
graphs the population dynamics of populations is needed, see app. IE.51 Moreover the 
reweighting taking in account the term (jj^i)"^ j s CO stly. It is thus extremely useful to 
check the local stability of the 1RSB solution in the lines of the appendix O The two 
types of local stability follow. 

• Stability of type I — the inter-cluster spin glass susceptibility does not diverge. 




"2 



kadi—j 



mp^ 1 })] , (d.i) 




(D.2) 



where the overline denotes an average over clusters 




(D.3) 
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• Stability of type II — the intra-cluster spin glass susceptibility does not diverge. 

The instability of second type is sometimes called the Gardner instability due to 
[Gar85j . 

Again, there are several equivalent ways how to investigate the 1RSB stability. This 
time we first describe the zero temperature - frozen fields - version before turning to the 
general formalism. 

D.l Stability of the energetic 1RSB solution 

In the energetic zero temperature limit the 1RSB distribution P l ~*i can be split 

into the frozen and soft part as in (14.21) . Moreover the self-consistency equations on the 
weights of the frozen fields, called the SP-y equations, do not depend on the details of the 
soft part. The methods for stability investigation of the SP-y equations were developed 
in [Par02bl IMRT081 IMPRT041 IRBMM04] . 

Type I — SP-y convergence — The divergence of the inter-cluster spin glass suscep- 
tibility is in general equivalent to the non- convergence of the 1RSB equations (12.241) on a 
single graph. The reason is exactly the same as for the equivalence of the non-divergence 
of the spin glass susceptibility and the convergence of the belief propagation equations, 
which we explained in app. IC.ll In the energetic zero temperature limit the convergence 
of the general 1RSB equations becomes convergence of the SP-y equations on a single 
graph. All the methods described in app. IC.ll for the stability of the belief propagation 
equations can be used directly. 

Remark in particular that the chain method (1C.3I) . used e.g. in [RBMM041 IKPW04] . 
is not the simplest choice. The chains of length d — > oo have to be considered numerically, 
and the treatable values are only d ~ 10 — 20. This leads to an imprecision for a relatively 
large numerical effort. It is much more precise to use for example the noise propagation 
(EJ) as e.g. in [ZDEB-3]. 

Type II — Bug proliferation — The intra-state susceptibility is investigated in 
exactly the same manner as the replica symmetric stability. The only difference is that 
the average over clusters have to be taken properly. The energetic 1RSB solution is 
based on the warning propagation equations averaged properly over the clusters. Thus 
the 1RSB stability of the type II leads to the bug proliferation, as in app. IC.2I averaged 
over the clusters. 

Roughly explained, if we consider a model with Ising spins, we have the three com- 
ponents surveys p = (p-,po,p + ) on each edge. Where p s is the probability over clusters 
that the warning on this edge takes the value s. Consider, as in app. IC.2I a warning u 
and one of the incoming warnings «o, the remaining incoming warnings are indexed by 
i = 1, . . . , k where k is distributed according to Q{k) (1C 14[) . Define Pk(a — > b\c — > d) 
as the probability, over clusters, that the warning u changes from a value a to a value 
b provided that the warning uq was changed from a value c to a value d. Consider 
Pk{a —>■ b\c — > d) as a matrix of dimension 6. And consider a chain of edges of length 
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d. The proliferation of an instability "bug" is given by the product of matrices Pk along 
this chain. The product is averaged over the realizations of disorder (in degrees, etc.). 
We define the stability parameter as 

\ II (d)= 1 {Tr(P 1 ki ...PQy« . (D.5) 

The SP-y is 1RSB stable if and only if lim^oo \n(d) < 1. For more detailed presentation 
of the 1RSB bug proliferation method or concrete examples see e.g. [RBMM04, MMZ06, 
IKPW04] and [ZDEB-3]. In all the implementations of this method the chain of d — > oo 
edges was used. Unlike in the type I stability, it is not know if this can be avoided in 
general. 



Some results — The investigation of the 1RSB stability as we just described can be 
very simply incorporated to the population dynamics method used to solve the survey 
propagation equations. This means that on random regular graphs the stability equations 
become algebraic, as the values of surveys do not depend on the index of the edge. In 
fig. ID. II we present the result for coloring of random regular graphs. 
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Figure D.l: The complexity as a function of energy for the coloring of random regular 
graphs. The 1RSB stable parts of the curves are in bold red. 

On all the parts of fig. ID. II the complexity function is plotted against energy, E(e) 
(I2.32j) . This function is the main output of the 1RSB energetic method, the SP-y equa- 
tions. The parameter y corresponds to the slope of the complexity function y = <9£(e) / de. 
Note that only the concave parts of the curves are physical. 

The red parts of the E(e) curves are the 1RSB stable parts. It seems to be a general 
fact that the instability of type I happens first for large values of y, and the instability 
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of the type II for small values of y. The unphysical (convex) branch is always type II 
instable. The instability of type I is sometimes completely absent. 

An important observation is that the stability of the 1RSB energetic solution does 
not guarantee the stability of the full 1RSB solution. Differently said, the soft fields can 
destabilize the full solution. On the other hand also the opposite is true — the instability 
of the clusters corresponding to m = does not imply the instability of the dominating 
clusters at m*. We thus want to stress that the results of [MPRT04L IRBMM041 IMMZ061 
IKPW04] and others have to be taken with these two facts in mind. 



D.2 1RSB stability at general m and T 

The stability of the full 1RSB equations at a general value of the parameter m and of 
the temperature T is a more difficult task. We are not aware of any study where this 
would be practically considered for models on random graphs, apart from [ZDEB-6]. We 
review shortly the main findings and difficulties. 



Type I — Divergence of the inter-cluster spin glass susceptibility ( ID. 21) is equivalent to 
the non-convergence of the probability distributions P l_ *- J (^ l_ * :? ) (12.241) . But here arrives 
the biggest problem, how to judge if a probability distribution converges? The probability 
distribution P l-+J '(^*~^) is represented by a population of random elements picked from 
this distribution. How to decouple the randomness coming from this sampling and the 
one coming from the eventual non- convergence? Of course, provided that the numerical 
difficulty does not rise to the level of directly solving the 2RSB equations. This is not 
known in general and it is a technical but important open problem in the subject. 

One interesting observation can be made, however: If the RS solution is instable then 
the 1RSB solution at m — 1 is type I instable. Indeed, if the mean value of the probability 
distribution does not converge then the 1RSB solution is type I instable. At the value 
m — 1 the mean ( 1A.3I) satisfies the simple belief propagation equations, as explained in 
app. \M 



Type II — Divergence of the intra-cluster spin glass susceptibility ( ID. 41) is much easier 
to investigate on a general level. It is equivalent to checking if the 1RSB iteration are 
stable against small changes in the probabilities ip- Arguably the simplest way to do 
so is the deviation of two replicas method, described for the RS stability in app. IC.ll 
We first find a fixed point of the 1RSB equations (12.241) using the population dynamics 
method. Then we create a second copy of the populations representing the distributions 
P l_> J '(-0 t- *J) . We perturb infinitesimally every of its elements ip l ^ J . The 1RSB is type II 
stable if and only if the two copies converge to the same point. The noise propagation 
and other methods from IC.ll can be used equivalently. 



Some results and connection to the SP-y stability — Fig. ID. 2l depicts the results 
for the stability of type II in the space of the parameters m and temperature T. The 
1RSB solution is type II stable above the red curve m n . 

It is interesting to state the connection between the general m, T stability and the 
energetic zero temperature limit. The parameter m = yT when T — > 0, thus when the 
stability of the frozen fields is relevant for the full stability the parameter yuT gives the 
slope of mii(T) near to zero T. This indeed seems to be the CcLSG, clS shown in fig. ID. 21 
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Figure D.2: Example of m-T diagrams for the 4-state anti-ferromagnetic Potts model 
on c-regular random graphs (left c = 10, right c = 13). A nontrivial solution of the 
1RSB eq. (12.241) exists above the curve m ex (green). The curves in blue m* represent the 
thermodynamic value of the parameter m. The red curve mn is the lower border of the 
type II stable region. The straight lines (ynT and y*T) represent the slopes corresponding 
to the energetic 1RSB solution yT = m in T — > 0. The energetic 1RSB solution is type 
II stable for y > yu- The line yuT seems to give correctly the slope of ran. This suggest 
that the stability of frozen variables is equivalent to the full stability for small m and T. 
Other examples of diagrams of this type are presented in [ZDEB-6]. 



Based on the arguments above, it seems reasonable that the following assumptions 
are correct: 

(i) The stability of the energetic method gives the full stability for small m and T. 

(ii) If the RS solutions is stable then the 1RSB is stable type I at m = 1. 

(iii) If the 1RSB at a given temperature is type I (II resp.) stable at a given m, then it 
is type I (II resp.) stable for all smaller (larger resp.) m. 

Assuming as above, the stability of the 1RSB solution in the region where the RS 
solution is stable is given by the type II (Gardner) stability, which we know how to 
investigate. The result is depicted e.g. in fig. 15.61 This would mean that the stability of 
type II is always more important for the thermodynamical solution. And in particular 
that in the random coloring problem for q > 4 the 1RSB solution is stable in all the 
colorable phase. 

The situation for 3-coloring is more subtle as 3-coloring is not RS stable for c > 
Cd- However, from assumption (i) follows that the interval of connectivities (c s , cq) = 
(4.69, 5.08) is 1RSB stable at small temperatures. Thus we expect also all the colorable 
phase to be 1RSB stable (otherwise the phase diagram at fig. 15.61 would have to present 
a sort of re-entrant behaviour). This would also be in agreement with the situation in 
the fully connected ferromagnetic 3-state Potts model [GKS85] 0. 



1 This is a contra-example to the common claim that in the systems with continuous dynamical 
transition (T<j = Ti oca i) the 1RSB solution is not stable. 
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Appendix E 
Populations dynamics 



Population dynamics is a numerical method to solve efficiently distributional equations 
of type fll.32p or (I2.24p and compute observables of type (11.331) or (I2.25aj) . In this context 
it was developed in [MP01J. As the form of the 1RSB equations was more or less known 
before, and they were solved approximatively using various forms of the variational ansatz, 
see e.g. |BMW00] . it may be argued that the population dynamics technique was the 
crucial ingredient which made the spin glass models on random graphs solvable. Recently 
rigorous versions of this method were developed to analyze the performance of decoding 
algorithms [R UOT] . the name density evolution is often used in this context. 

The main idea is to represent the probability distribution by a population (sample) 
of iV elements drawn independently at random from this distribution. The algorithm 
starts from a random list and it mimics T iterations of the distributional equations and 
(hopefully) converges to a good representation of the desired fixed point. Several gener- 
alizations or subtleties are encountered and we describe some of them in the following. 
Consider the a random constraint satisfaction model specified by degree distribution 
IZ(k) of constraints, and Q{1) of variables, the excess degree distributions r{k) and q{l) 
are given by (jl.8p . 

E.l Population dynamics for belief propagation 

The simplest version of the population dynamics is used to solve 

• Belief propagation distributional equations fll.23atil.23bj) and compute the corre- 
sponding average free energy (ll.20p . entropy, etc. The complete replica symmetric 
solution is obtained this way. 

• Survey propagation distributional equations, obtained from (jl.41til.42j) . and com- 
pute the average complexity function fll.43j) . The satisfiability transition is obtained 
this way. 

The pseudocode for the procedures Population- Dynamics and One- Measurement 
follows. To compute the observable $ (free energy, entropy, complexity, etc.) we first 
call procedure Population- Dynamics with T = T eqU ii (equilibration time) and suffi- 
ciently large N. After we repeat One- Measurement plus Population- Dynamics 
with T = T ranc j (randomization time) and M sufficiently large, but smaller than N. And 
finally we compute averages and error bars of these measurements. 
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In some problems the constraints are themselves random (negations in i^-SAT, in- 
teractions in a spin glass etc.). The choice of this quenched randomness is then done at 
line M of Population- Dynamics, and at line [8] of One-Measurement. 

The population {?/>} is randomly initialized to a random assignment at line Q] of 
Population- Dynamics. That is all the zero components of the surveys (jl.41til.42l) 
are zero, and the beliefs are completely biased, i.e., either (1,0) or (0, 1). Such a choice 
is justified from the analogy with the reconstruction on trees where the proper initial 
condition is given by (12.111) . 

Satisfactory results are usually obtained with the population sizes and times of order 
N « 10 4 - 10 5 , T equil « 10 3 - 10 4 , T rand « 10, M « N. But these values may change 
problem from problem and a special care have to be taken about the numerics every time 
as basically no convergence theorems are known for a general case. 

Population- Dynamics (r(/c), q(l), N, T) 

1 Initialize randomly iV-component array {ip}; 

2 for t = 1,...,T: 



3 do for i = 1, ...,N: 

4 do Draw an integer k from the distribution r(k); 

5 for d — 1, . . . , k: 

6 do Draw an integer I from the distribution q(l); 

7 Draw indexes ji, ■ ■ ■ ,ji uniformly in {1, . . . , iV}; 

8 Compute Xd from . . . , if)j t } according to eq. ( j!.16bl) ; 

9 Compute V'new from {xi, ■ ■ ■ ,Xk} according to eq. ( I1.16al) ; 

10 ipi «- Vv w ; 



11 return array {ip}; 
One-Measurement(^(A;), Q(l), q(l), N, M) 



1 Initialize Constraint = 0; $ variable = 0; 

2 for i — 1, . . . , M: > Compute the constraint part. 

3 do Draw an integer k from the distribution lZ(k); 

4 for d = 1, . . . ,k: 

5 do Draw an integer I from the distribution q(l); 

6 Draw indexes ji, ■ ■ ■ ,ji uniformly in {1, . . . , iV}; 

7 Compute Xd = Iln=iV'i„; /r ^ m 

8 Compute Z new from {xi, • • • , Xk} according to eq. (11.19aj) ; 

9 ^constraint ^ ^"constraint log Z neW) 

10 for i = 1, . . . , M: > Compute the variable part. 

11 do Draw an integer I from the distribution Q(l); 

12 Draw indexes ji, ■ ■ ■ ,ji uniformly in {1, . . . , iV}; 

13 Compute Z new from {i/j^, . . . , ipj t } according to eq. (jl,19bj) ; 

14 "^variable <~ ^variable + (I — 1) \ogZ ncw ] 

15 return (a$ const raint - $variablc)/M; 



E.2 Population dynamics to solve 1RSB at m = 1 



The general 1RSB equations for general random graph ensemble require a population 
dynamics with population of populations. We will explain this in sec. IE. 51 Treating 
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the population of populations requires a lot of CPU time and it is not very precise, thus 
anytime we have the opportunity to avoid this we have to take it. One such opportunity is 
the simplification of the 1RSB equations at m — 1 explained in appendix^] Conveniently, 
both the clustering and the condensation transitions are obtained this way. 

The population dynamics method have to be adapted to solve eq. (j A. 10ft and to 
measure the entropy of states ( lA.llj) . We give the m = 1 generalization of the procedure 
Population- Dynamics, the changes in One-Measurement are then straightforward. 
Note that lines [11] and [13] take in general 2 k steps as we need to compute probability of 
every combination of the set {si, . . . , s^}. 

PD-(m = l)-GENERALIZATION(r(fc), g(7), iV, T) 

1 {ip RS } <— Population- Dynamics (r( A;), q(l), N, T); 

2 Initialize iV-component arrays {ip 1 <— 1} and {ip° ^ 0}; 

3 for t = 1,...,T: 

4 for i = 1, . . . , JV: 

5 do Draw an integer k from the distribution r(k); 

6 for d — 1, . . . , k: 

7 do Draw an integer Id from the distribution q(l); 

8 Draw indexes j(d, 1), . . . ,j(d, Id) uniformly in {1, . . . , iV}; 

9 Compute %d from {ipf^ i)j • • • j i d )} according to eq. (11.16bj) ; 

10 s<-l; 

11 Choose {s\, . . . , 5^} with prob. given by the 2 nd line of eq. (TOOl) ; 

12 s^O; 

13 Choose {ri, . . . , rfc} with prob. given by the 2 nd line of eq. ( 1A.10I) ; 

14 for d = 1 fc: 



15 do Compute from 1)5 • • • 5 ^j^/^)} according to eq. (11.16b)) ; 

16 Compute \°d from {ipjf d iy ■ ■ ■ , ^jtdi d )} according to eq. fll.l6bj) ; 

17 Compute ^„ e s w from {xf s , • • • , xf S } according to eq. ( 11.16a)) : 

18 Compute ip\ evf from {x\, • • • , x\} according to eq. (II . 16aj) : 

19 Compute ip^ ew from {xl, ■ ■ ■ , xVi according to eq. (]1.16aj) ; 

20 

21 ^ - ^ ew ; 

22 ^ <- < ew ; 

23 return arrays {V' RS }, {ip 1 }, {ip }', 

E.3 Population dynamics with reweighting 

A simplification of the 1RSB equations ( 12.241) arises for the ensemble of random regular 
graphs, there the distribution V l ^{ip' 1 ^) over clusters is the same for every edge (ij). 
In the corresponding population dynamics a special care have to be taken about the 
reweighting term (Z*^- 7 )" 1 . 

We describe two different strategies to deal with the reweighting. In the first one 
Reweighting- Faster the elements of the population have all the same weight and thus 
in each sweep the population needs to be re-sampled and some less probable elements 
might be lost. In the second strategy Regular-Reweighting-Precise each element 
has its own weight, no re-sampling is needed, but the search of a random element, at the 
line HOl takes log steps. Thus the first strategy is faster, the second one is slightly more 
precise. Which one is eventually better seems to be problem specific. 
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Consider a population {ip} where each element ipi has weight W{. The weights are 



computed from the BP update ( 11. 16atfT7l6ajl as Wj = ( Z a ^ % fX/e^a-i Z^ 




Reweighting-Faster(JV, {ip}, {w}) 

1 Wtat <— 0; 

2 for z = l,...,JV: 

3 do w tot <— w tot + wf, 

4 > Zi is the cumulative distribution of indexes z; 

5 z = 0; 

6 for i = 1, . . . , N: 

7 do Zi <— + Wi/w tot 

8 > Trick to make a list of ordered random numbers rij in O(N) steps. 

9 G «- 0; 

10 for z = 1, . . . , JV: 

11 donj < log Rand; 

12 > RAND outputs a random number in the interval (0, 1). 

13 G <- G + rii, 

14 G <— G - log Rand; 

15 ni <— ni/G] 

16 for i = 2, . . . , N: 

17 do rij «— rii/G; 

18 rij «- ni + nj_i; 

19 > Finally making the new population. 

20 p<-0; 

21 for z = 1, . . . , JV 

22 do while (jii > z p ) p <— p + 1; 

23 C ew ^ P ; 

24 return array {^ new }; 

Regular- REWEiGHTiNG-PRECiSE(r(fc), q(l), JV, T, m) 

1 Initialize randomly JV-component arrays {ip} and {w}; 

2 for t = 1, . . . , T: 

3 for i = 1,..., JV: 



4 do Draw an integer from the distribution r(fc); 

c 7 , i . 

6 for d = 1, . . . , k: 

7 do Draw an integer I from the distribution q(l); 

8 for n — 1, . . . , I: 

9 do Create cumulative probability distribution from weights {w}; 

10 Draw index j n from this cumulative distribution; 

11 Compute Xd from {i/j^, . . . ,ipj t } according to eq. ( 11.16bl ); 

12 Z new <— Z ncw ■ Zd, where Z d is the norm, from eq. ( 11.16bl) ; 

13 Compute ipnew from {xi, ■ ■ ■ , Xk} according to eq. ( 11.16a!) ; 

14 Z new <— Z ncw ■ Zd, where Z d is the norm, from eq. (I1.16a|) ; 

16 ^ <- (2new) m ; 



17 return array -f^}, weights {^}; 
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E.4 Population dynamics with hard and soft fields 

Fraction of frozen variables (again on random regular graphs for simplicity) can be ob- 
tained by solving equation (14.101) . To compute the value r(ra) a population needs to be 
kept for the soft part of the distribution P so ft, eq. (14. 2p . It is important to stress that 
when evaluating the if conditions on lines [T5imJ1 and [23] we consider as frozen only the 
incoming fields created at line [T51 

PD-HARD-SOFT(r(£;), q(l),N, T, m) 

1 Initialize randomly iV-component array {ip <— Rand}; 

2 Tj<-1; 

3 for t = 1,...,T: 



4 do % <— 1; 

5 h <— 0; Zhard 0; Z so f t <— 0; 

6 while i < N: 

7 do Draw an integer k from the distribution r(k); 

8 Z ncw < 1, 

9 for d = 1, . . . , k: 

10 do Draw an integer I from the distribution q(l); 

11 for r = 1, . . . , I: 

12 do if Rand < 77 

13 then Set ip r to be a frozen field; 

14 else Draw ip r uniformly from {ip}; 

15 if No contradiction between the frozen fields in {ipi, ...,?/>;} 

16 then Compute Xd from {?pi, . . . , ipi} using eq. fll.l6bjl : 

17 Z ncw <— Z ncw ■ Z d , Z d is the norm, from (jl.l6b[) : 

18 else goto line El 

19 if No contradiction between the frozen fields in {xi, ■ ■ ■ , Xk} 

20 then Compute ip new from {xi, ■ ■ ■ ,Xk} according to eq. (II . 16aj) : 

21 Z new <— Z ncw ■ Zd, where Zd is the norm, from eq. (11.16aj) ; 

22 else goto line [TJ 

23 if ip new is a frozen field 

24 then Zhard -^hard + (^new) ] 

25 + 

26 else Z so { t <— Z so f t + (Z new ) ; 

27 Ipi <- ^ new ; 

28 Wi <- (Z new ) m ; 

29 i<-i + l; 

30 r <- (Z soft /i)/(Z hard iV); 

31 Update 77 according to eq. (14-lOj) : 

32 {ip} <— REWEIGHTING-FASTER(iV, {^}, {w}); 



33 return array {ip}, rj; 

E.5 The population of populations 

The general 1RSB equations take form (I2.33p . the order parameter V[P(if))] is a distribu- 
tion (over the graph ensemble) of distributions (over the clusters). It can be represented 
by a population {{^}} of iV-component populations where i = 1,...,M. We 
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sketch here the corresponding population dynamics of populations. Again this has been 
first described in |MP01j . 

POPULATION-OF-PoPULATIONS(r(/;;), q(l), N, M, T, m) 

1 Initialize randomly M x iV-component array {{"0}}; 

2 for t = 1,...,T: 

3 do for i = 1, . . . , M: 

4 do Draw an integer k from the distribution r(k); 

5 for d — 1, . . . , k: 

6 do Draw an integer from the distribution q(l); 

7 Draw indexes i(d, 1 ),..., i(d, Id) uniformly in {1, . . . , M}; 

8 {^}new <- One-Step({{^}}, {z(l, 1), . . . , i(k, l k )}, {I}, k, N, m); 

9 <— {^}new; 

10 return array {{^}}; 

One-Step({{^}}, {i(l, 1), . . . , i(k, l k )}, {I}, k, N, m) 

1 for./ 1 V: 

2 do Z ncw 4- 1; 

3 for d — 1, . . . , k: 

4 do Draw indexes j(d, 1), . . . , Z^) uniformly in {1, . . . , N}; 

5 Compute \d from {^i( d ,i),j(d,i), • • • , ),j(d,i d )} using (|1.16b|); 

6 Z ncw <— Z ncw • Zrf, Z<i is the norm, from ( 11.16bl) ; 

7 Compute ^ new from {xi, . . . , Xk} according to eq. (II . 16ei[) : 

8 Z new <— Z ncw • Zd, is the norm from (II . 16ei[) : 

9 Wj <— (Z ncw ) m ; 

10 Ipj <- V'new; 

11 {■?/>} <— Reweighting-Faster(./V, {■?/>}, {w}); 

12 return array {V'}; 

Depending on the problem we are about to solve the population of populations might 
also be combined with the reweighting of populations or the separation of the frozen and 
soft fields, see e.g. appendix D of [ZDEB-5]. 



E.6 How many populations needed? 

We make a summary of which level of the population dynamics technique is needed 
depending on the problem. References are just examples and are biased towards works 
presented in this thesis. 

• Analytical solution 

— Belief propagation on regular graphs [ZDEB-1, ZDEB-5, ZDEB-9]. 

— General warning propagation with integer warnings [ZDEB-1, ZDEB-3]. 

— Frozen variables at m = 1 [ZDEB-5, ZDEB-9]. 

— Survey propagation on regular graphs (frozen variables at m = 0, energetic 
cavity) [KPW04] or [ZDEB-5, ZDEB-9]. 
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• Single population 

— General belief propagation in models with discrete variables [ZDEB-1, ZDEB- 
5, ZDEB-9]. 

— General survey propagation (1RSB at m = 0, energetic cavity) on model with 
integer warnings [ZDEB-3, ZDEB-9], or very precise numerics in |MMZ06j . 

— 1RSB at m = 1 (MM06al IMRTS08] or [ZDEB-4, ZDEB-5]. 

— 1RSB on random regular graphs [ZDEB-4, ZDEB-5]. 

— 2RSB at m = (energetic cavity) on regular graphs [Riv05] . 

• Population of populations 

— General 1RSB (also finite temperature) jMPOR IMPR05} IMRTS08| or [ZDEB- 
4, ZDEB-5, ZDEB-6]. 

— 2RSB of random regular graphs [ZDEB-6]. 

— 2RSB at m = (energetic cavity). 

— 3RSB at m = (energetic cavity) on regular graphs. 

We are not aware on any work where the last two points would be implemented. More 
levels of replica symmetry breaking would require more levels of populations. We are not 
aware of any work where more than population of populations would be treated. Rather 
than pushing the numerics in this direction new theoretical works are needed for models 
where the 1RSB solution is not correct. 
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Appendix F 
Algorithms 



Here we do not aim to provide a complete summary of algorithms used to solve the 
random constraint satisfaction problems. We just define and briefly discuss algorithms 
which were used, generalized or tested in the context of this thesis. Strictly speaking we 
are almost always dealing with incomplete solvers, that is algorithms which might find a 
solution but never provide a certificate of unsatisfiability. It is an open and interesting 
questions if the methods presented in this thesis can imply something for certification of 
unsatisfiability. 

F.l Decimation based solvers 

A large class of algorithms for CSPs is based on the following iterative scheme: 
Decimation 

1 repeat Choose a variable i ; 

2 Choose a value Sj ; 

3 Assign % the value Sj and simplify the formula; 

4 until Solution or contradiction is found; 

The nontrivial part is how to choose a variable in step ED and how to choose its value 
in step El In the following we describe several more or less sophisticated or efficient 
strategies. 

Note that all these strategies can be improved by backtracking, that is if a contradiction 
was found we return to the last variable where another value than the one we chose was 
possible and make this choice instead. 

F.l.l Unit Clause propagation 

One of the simplest (and obvious) strategies is to choose and assign a variable which 
is present in a constraint which is compatible with only one value of that variable. In 
K-SAT this is equivalent to assigning variables belonging to clauses which contain only 
this variable, hence the name unit clause. If no such variable exists one possibility (the 
random heuristics) is to choose an arbitrary variable and assign it a random value from 
the available ones. The unit clause propagation combined with the random heuristics 
(without backtracking) is not very efficient solver of K-SAT. But the situation is more 
fortunate for some other constraint satisfaction problems. The most interesting example 
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being perhaps the 1-in-K SAT [ACIMOl] and [ZDEB-3]. The random 1-in-K SAT exhibits 
a sharp satisfiability phase transition for K > 3. Moreover, if the probability of negation 
of variables lies in the interval (0.2726, 0.7274) (for K = 3) then: 

• In the satisfiable phase the unit clause propagation combined with the random 
heuristics finds a solution with finite probability in every run. 

• In the unsatisfiable phase every run of the unit clause propagation leads to a con- 
tradiction with finite probability after the assignment of the very first variable. 

Hence, with random restarts the random l-in-3 SAT is almost surely solvable in poly- 
nomial time in the whole phase space (given the probability of a negation is as specified 
above). At the same time the l-in-3 SAT is an NP-complete problem, it thus provides 
a rare example of an on average easy NP-complete problem with a satisfiability phase 
transition. 

Unit clause propagation is the main element of all the exact solvers of constraint 
satisfaction problems. The most studied example being the Davis-Putnam-Logemann- 
Loveland (DPLL) algorithm [DPoH, IDLL62j for K-SAT which combines the unit clause 
propagation with the pure literal elimination (pure literal appears either only negated or 
non-negated) with backtracking. It was mostly this algorithm which was used when the 
connection between the algorithmical hardness and phase transitions was being discovered 
|MS L92[ ICKT91j . Moreover, all the modern complete solvers of the satisfiability problem 
follow a similar, more elaborated, path. 

F.1.2 Belief propagation based decimation 

Belief propagation |Pea82j computes, or on general graphs approximates, marginal proba- 
bilities. These can then be used to find an actual solution. In some problems the marginals 
give the solution directly, e.g. in the error correcting codes |Gal68] . in the matching 
[BSS05, BSS06J, or the random field Ising model at zero temperature [KW051 ICheOSj 
etc. In constraint satisfaction problems, typically, marginals do not give a direct infor- 
mation about a solution. For example in coloring of random graphs, the BP equations 
always converge to all marginals being equal to 1/q. Belief propagation based decimation 
strategies have been studied recently. 

In every cycle of the algorithm Decimation, the belief propagation equations are 
updated until they converge or a maximal number of updates per variable T max is reached. 
At least two strategies how to choose the decimated variable and its value were tested 
and studied, see e.g. [ZDEB-4] and |MRTS07j : 

• Uniform BP decimation - Choose a variable at random and assign its value accord- 
ing to the marginal probability estimated by BP. 

• Maximal BP decimation - Find the variable with the most biased BP marginal and 
assign it the most probable value. 

The other two combinations where a random variables is assigned its most probable value 
or when the most biased variable is assigned random value according to its marginal prob- 
ability can be think of. The BP decimation, as described above, runs in quadratic time. 
In eventual practical implementations a small fraction of variables should be decimated 
at each step, thus reducing the computational complexity to linear (or log-linear if the 
maximum convergence time increases as log A). 
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The empirically best strategy is the maximal BP decimation. This can be understood 
from the fact that this strategy aims to destroy the smallest possible number of solutions 
in every step, as argued on a more quantitative level in |Par03j . We gave as an example 
the performance of the maximal BP decimation in the 3- and 4-coloring of random Erdos- 
Renyi graphs [ZDEB-5] in fig. 13.31 

The uniform BP decimation is less successful, because it aims not only to find a 
solution but also to sample solutions uniformly at random. Indeed, if an exact calculation 
of marginal probabilities would be used instead of the BP estimates the uniform exact 
decimation would lead to a perfect sampling. The uniform exact decimation is a process 
which can be analyzed using the cavity method. The result then sheds light on the 
limitations of the BP decimation. This analysis was developed in [MRTS07J, and we give 
an example for the factorized occupation problems in the following. 

Maximal BP decimation on the random coloring 

We implemented the maximal BP decimation algorithm on the random graph coloring. 
We chose T max = 10, if a solutions is not found we restart with T max = 20 and eventually 
once again with T max = 40. The fraction of successful runs is plotted in fig. 13.31 and we 
see that this algorithm works even in condensed phase where the BP marginals are not 
asymptotically correct, or in a phase where the equations do not even converge. The 
non-convergence of the belief propagation equations is ignored (in 3-coloring from the 
beginning, in 4-coloring after a small fraction, typically around 10%, of variables was 
fixed). It thus seems that in coloring the BP decimation is a very robust algorithm. 

What is the reason for the failure of the maximal BP decimation at higher connectiv- 
ities? A straightforward suggestion would be that is should not work in the condensed 
phase where the BP marginals are not asymptotically correct. But we do not observe 
anything particular in the performance curves at the condensation transition. A second 
natural suggestion would be that BP should converge in order that the algorithm works, 
this also does not seem to be the case, as BP does not converge in the 3-coloring for 
connectivity c > 4 and yet the algorithm is perfectly able to find solutions. Moreover, 
even in 4-coloring where the BP equations converge on large formulas in all the satisfiable 
phase, after a certain (rather small) fraction of variables is decimated the convergence is 
lost. As we argued in appendix O the non- convergence of BP is equivalent to the local 
instability of the replica symmetric solution. It thus seems that the reduced problem, 
after a certain fraction of variable was fixed, is even harder from the statistical physics 
perspective than the original problem. Yet, this does not seem to be fatal for the finding 
of solutions. Finally, in the region where the BP decimation algorithm really does not 
succeed we observed that a precursor of the failure exists. The normalizations in the BP 
equations fll.16aHl.16bl) gradually decreases to zero, meaning that the incoming beliefs 
become almost contradictory. 

Analysis of the uniform exact decimation 

The uniform exact decimation after ON steps is equivalent to taking a solution uniformly 
at random and fixing its first ON variables. Such a procedure can be analyzed [MRTS07J 
and conclusions made about the influence of small errors in the BP estimates of marginals. 

Given an instance of the CSP, consider a solution {s} taken uniformly at random and 
reveal the value of each variable with probability 0. Denote $ the fraction of variables 
which were either revealed or are directly implied by the revealed ones. To compute $(#) 
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we derive the cavity equations on a tree. Denote the probability that a variable i is 
fixed conditioned on the value s of the variable i and on the absence of the edge (ib): 



= + (l - 0) 



(F.l) 



a£di—b 

Meaning that the variable i was either revealed or not, and if not it is implied if at least 
one of the incoming constraints implies it. The q^ 1 is a probability that constraint a 
implies variable % to be s conditioned on: 1) variable i taking the value s G {s} in the 
solution we chose, 2) variable i was not revealed directly and 3) the edge (cm) is absent. 

We write the expression for q^ 1 only for random occupation CSPs on random regular 
graphs where the replica symmetric equation is factorized. Then also q^ 1 and <3>*^ 6 are 
factorized, that is independent of a,b,i. The conditioned probability q s is the ratio of 
the probability that variable % takes the value s and is implied by the constraint a and 
probability that variable % takes the value s: 



qi 



go 



E (T) (^) lr M {k ~ r) E So-^r (1 - ^i) S , (F-2a) 
A r+1 =l,A r =0 ^ ' s=o \ ' 

E Q i^) lr M {k - r) E (* 7) 0- ~ $ o)%F.2b) 



ru A r =l )J 4 r+ i=0 x 7 s=0 



where / = L — 1, k = K — l. The indexes s\, Sq in the second sum of both equations are the 
largest possible but such that s x < r, s < K — 1 — r, and XlsLo ^r-s = 0, J2T=o A r +i+s = 
0. The terms $;$£- r-8 (l - $ ) s and $^ s $^- r - 1 (l - are the probabilities that a 
sufficient number of incoming variables was revealed such that the out-coming variable is 
implied (not conditioned on its value). The first sum goes over all the possible numbers of 
l's being assigned on the incoming variables, r. The term ip l {ipQ k r ^ is then the probability 
that such a configuration took place. The cavity probabilities that the corresponding 
variable takes value 0/1, ipo,ipi are taken from the BP equations fl4.16ati4.16bp . Z reg is 
the normalization in fl4.16aH4.16bp . The first condition on r takes care about the values 
of the incoming neighbours being compatible with the value of the variable % on which is 
conditioned, the second condition on r is satisfied if and only if the value of the variable 
% is implied by the incoming configuration. 

Once a solution for q s is found (from initial conditions $ = 9) the total probability 
that a variable is fixed is computed as 

$(0) =9 + (l-9) {^[1 - (1 - Ql ) L ] + // [1 - (1 - go) L ]} , (F.3) 

where /i , H\ are the total BP marginals, \i s = ips/i^o + V'f )■ 

Notice the complete analogy between eqs. flF.2bHF.2aj) and the equations for hard 
fields at m = 1 fl4.20bH4.20al) . To compute the function $(#) for a general CSP on 
a general graph ensemble a derivation in the lines of app. |A] have to be adapted, see 
also [MRTS07]. Finally note that as the probabilities ipo are taken from the belief 
propagation equations the form flF.2bHF.2~aj) is not correct in the condensed phase (but 
in the locked problems the satisfiable phase is never condensed). 

The Failure of Decimation in the Locked problems 

In the locked problems, see sec. 14.31 the BP decimation algorithm does not succeed to find 
a satisfying assignment even at the lowest possible connectivity. To give an example in 
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the l-or-3-in-5 SAT on truncated Poissonian graphs the maximal BP decimation succeeds 
to find a solution in only about 25% at the lowest average connectivity 1 = 2, and this 
fraction drops down to less than 5% at already I = 2.3 (to be compared with the clustering 
threshold Id = 3.07, or the satisfiability threshold l s = 4.72). 

Interestingly, the precursors of the failure of the BP decimation algorithm observed in 
the graph coloring are not present in the locked problems. In particular the BP equations 
converge during all the process and the normalizations in the BP equations (ll.16aHl.16bj) 
stays finite. However, the above analysis of the function sheds light on the origin of 
the failure. 
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Figure F.l: Analytical analysis of the BP inspired uniform decimation. Number of vari- 
ables directly implied plotted against number of variables fixed 9. 



In fig. IF. II we compare the function (IF. 3[) with the experimental performance of 



the uniform BP decimation. Before the failure of the algorithm (when a contradiction 
is encountered) the two curves collapse perfectly. The reason why the algorithm fails to 
find solutions is now transparent. 

• Avalanche of direct implications - In some cases the function has a disconti- 
nuity at a certain spinodal point 9 S (6 S « 0.46 at L = 3 of the l-or-3-in-5 SAT). 
Before 9 S after fixing one variable there is a finite number of direct implications. As 
the loops are of order log N these implications never lead to a contradiction. At the 
spinodal point 9 S after fixing one more variable and extensive avalanche of direct 
implications follows. Small (order 1/N) errors in the previously used BP marginals 
may thus lead to a contradiction. This indeed happens in almost all the runs we 
have done. For more detailed discussion see [MRTS07J. 

• No more free variables - The second reason for the failure is specific to the locked 
problems, more precisely to the problems where $ = 1 is a solutions of flF.2arlF.2bl) . 
In these cases function $(9) — > 1 at some 9\ < 1 (9i « 0.73 at L = 4 of l-or-3-in-5 
SAT). In other words if we reveal a fraction 9 > 9\ of variables from a random 
solution, the reduced problem will be compatible with only that given solution. 
Again a little error in the previously fixed variables and the BP uniform decimation 
ends up in a contradiction. If on the contrary the function §(9) reaches value 1 
only for 9 = 1 then the residual entropy is positive and there should everytime be 
some space to correct previous small errors, demonstrated on a non-locked problem 
in fig. ES 
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These two reasons of failure of the BP uniform decimation seems quite different. But 
they have one property in common. As the point of failure is approached we observe a 
divergence of the ratio between the number of variables which were not directly implied 
before being fixed and the number of directly implied variables, see fig. IF.21 This ratio 
can also be computed for the maximal BP decimation and no quantitative difference is 
observed for the locked problems, thus the two reasons above explain also the failure of 
the, otherwise more efficient, maximal BP decimation. 
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Figure F.2: Left: For comparison, the BP uniform decimation works well on the non- 
locked problems, the example is for bicoloring. Right: Comparison of the maximal and 
uniform decimation. Number of directly implied variables is plotted against number 
of variables which were free just before being fixed. Behaviour of the two decimation 
strategies is similar. 



F.1.3 Survey propagation based decimation 

The seminal works [MPZ02, MZ02J not only derived the survey propagation equations, 
but also suggested it as a base for a decimation algorithm for random 3-SAT. The perfor- 
mance is spectacular, near to the satisfiability threshold on large random 3-SAT formulas 
it works faster than any other known algorithm. SP based decimation seem to be able 
to find solutions in 0(N log N) time up to the connectivity a = 4.252 in 3-SAT |Par03j 
(to be compared with the satisfiability threshold a s = 4.267). 

Survey propagation equations (11. 41111 .421) aim to compute the probability (over clus- 
ters) that a certain variables is frozen to take a certain value. This information can then 
be used to design a strategy for the Decimation algorithm. In particular, as long as 
the result of survey propagation is nontrivial (not all p % ^* a = 1) the variable with the 
largest bias \p\ — p l _\ is chosen and is assigned the more probable value. After a certain 
fraction of variables is decimated the fixed point of the survey propagation on the reduced 
formula is trivial. The suggestion of [MPZ02, MZ02] is that such a reduced formula is 
easily satisfiable and some of the well known heuristic algorithms may be used to solve it 
(Walk-SAT, see the next section lF~2~2l was used in the original implementation). Note 
also that the original implementation of [MPZ02, MZ02J decimated a fraction of variables 
at each Decimation step, thus reducing significantly the computational time. 
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Originally, the success of the survey propagation based algorithm was contributed to 
the fact that survey propagation equations take into account the clustering of solutions. 
This was, however, put in doubt since. To give an example, in the locked problems, 
see sec. I4.3j the survey propagation equations give an identical fixed point as the belief 
propagation and as we argued in the previous section IF. 1 .21 the maximal BP decimation 
fails to find solutions in the locked problem in the whole range of connectivities. 

The true reason for the high performance of survey propagation in 3-SAT thus stays 
an open problem. For example, and unlike with BP, there are usually no problems with 
SP convergence during the decimation. Two very interesting observations were made in 
[KSS07aJ for SP the decimation algorithm on K-SAT. First, the SP decimation indeed 
makes the formula gradually simpler for local search algorithms, see sec. IF.2.2|. again in 
contrast with BP decimation. Second, the SP decimation on i^-SAT does not create any 
(or a very small number) of direct implications (unit clauses) during the process. Given 
that creation of direct implication makes the decimation fail in the locked problems, as 
we just showed, this might be a promising direction for a new understanding. 



F.2 Search of improvement based solvers 

Here we describe another large class of CSPs solvers, the search of improvement algo- 
rithms. All these algorithms start with a random assignment of variables. Then different 
rules are adopted to gradually improve this assignment and eventually to find a solu- 
tion. The most typical example of that strategy is the simulated annealing [KGV83j or 
stochastic local search algorithms like Walk-SAT |SLM92l ISKC94] . 



F.2.1 Simulated annealing 

In physics simulated annealing is a popular and very universal solver of optimization 
problems. It is based on running the Metropolis [M RR+53j (or other Monte Carlo) 
algorithm and gradually decreasing the temperature-like parameter. Simulated annealing 
algorithm respects the detailed balance condition, after large time it thus converges to 
the equilibrium state, and it is thus guaranteed to find the optimal state in a finite time 
for a finite system size. In general, the time can of course depend exponentially on the 
system size, and in such it is not really of practical interest. 

We argued in chap. [2] that at the clustering (dynamical) transition the equilibration 
time of a detailed balance local dynamics diverges. However, the clusters which appear at 
the dynamical energy E d > have bottom at an energy ^bottom < E d and numerical per- 
formance of the simulated annealing in the 3-coloring of random graphs [vMS02j suggests 
that -^bottom might be zero even if E d is positive. More precise numerical investigation of 
this point is, however, needed. 



F.2. 2 Stochastic local search 

Solving K-SAT by a pure random walk was suggested in |Pap91| : 
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Pure- Random- WALK-SAT(T max ) 

1 Draw a random assignment of variables; 

2 T «- 0; 

3 repeat Draw a random unsatisfied constraint a; 

4 Flip a random variable i belonging to a; 

5 T<-T+l; 

6 until Solution is found or T > AT max ; 



In random 3-SAT this simple strategy seems to work in linear time up to «rw ~ 2.7 
[SM03j . Improvements of the Pure- Random- Walk-SAT have led to a large class 
of so-called stochastic local search algorithms. All are based on a random walk in the 
configurational space with more complicated rules about which variables would be flipped. 
The version called WalkSAT introduced in [SKC941ISKC96] became, next to the DPLL- 
based exact solvers, the most widely used solver of practical SAT instances. In random 3- 
SAT the Walk-SAT with p = 0.5 was shown to work in linear time up to about a\ys — 4.15 
|A(4K04] . 



WalkSAT (T max ,p) 

1 Draw a random assignment of variables; 

2 T <- 0; 

3 repeat Pick a random unsatisfied constraint a; 



4 if Exists a variable i in a that is not necessary in any other constraint; 

5 then Flip this variable i; 

6 else if Rand < p; 

7 then Flip a random variable % belonging to a; 

8 else Flip % (from a) that minimizes the # of unsat. constraints; 

9 T«-T + l; 

10 until Solution is found or T > AT max ; 



Several other variants of stochastic local search on random 3-SAT were studied in 
[SAO0 5] showing that with a proper tuning of parameters like p the linear performance 
can be extended up to at least a ~ 4.20. Finally a version of the stochastic local search 
called AS AT was introduced in [AA06] . In random 3-SAT AS AT works in a linear time 
at least up to a = 4.21 [A A O 6] . We adapted the implementation of AS AT and studied 
its performance in coloring and on the occupation CSPs. 
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ASAT(T max ,p) 

1 Draw a random assignment of variables; 

2 T <- 0; 

3 Create the list {f } of variables which are present in unsatisfied constraints. 

4 repeat Pick a random variable % from the list {v}; 



5 Compute the change of energy AE if the value of i is flipped. 

6 if AE < 0; 

7 then Flip i\ 

8 else if Rand < p; 

9 then Flip i\ 

10 else Do nothing; 

11 Update list {v} of variables which are present in unsatisfied constraints. 

12 T«-T+l; 

13 until Solution is found or T > NT max : 



In the coloring problem where variables take one from more than two possible values, 
the only modification of AS AT is that we choose a random value into which the variable is 
flipped on line El The performance for the 4-coloring of Erdos-Renyi graphs was sketched 
in fig. El 

There are two free parameters in the ASAT algorithm, the maximal number of steps 
per variable T max and, more importantly, the greediness (temperature-like) parameter p, 
which need to be optimized. In [AA06J and [ZDEB-5] it was observed that in the random 
K-SAT and random coloring problems the optimal value of p does not depend on the 
system size N, neither very strongly on the constraint density a. But these observation 
might be model dependent, as it indeed seems to be the case for the locked problems. 



F.2.3 Belief propagation reinforcement 

A " search of improvement" solver can also be based on the belief propagation equations. 
The idea of the belief propagation reinforcement, introduced in |CFMZ05] 3, is to write 
belief propagation equations with an external "magnetic" field (site potential) p,\. 



<r = ^ E n xir> ( F - 4a ) 

xir a = ^< n <*> ( R4b ) 

and then iteratively update this field in order to make the procedure converge to a 
solution given by the direction of the external field = argmax Sj /i* .. At every step 
the configuration given by the direction of the external field is regarded as the current 
configuration which is being improved. 

The question is how to update the external field. The basic idea is to choose the 
local potential fi\. in some way proportional to the current value of the total marginal 
probability xl^ which is computed without the external fields as 

£ = yU <* ■ ( R5 ) 

bedi 



1 Strictly speaking the reinforcement strategy was fist introduced for the survey propagation equations, 
but the concept is the same for belief propagation. 
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How exactly, and how often should the value of local potential be updated is open to 
many different implementations, some of them can be found in [BZ06, DRZ08]. The same 
as in the local search algorithm it is not well understood, beyond a purely experimental 
level, how the details of the implementation influence the final performance. We tried 
several ways and the best performing seemed to be the following 

/4 = (7r)^\ /4 = (l-7r)^\ if £o>£L (F.6a) 
^ = (1-^-1, // = (7r)^\ if Q<€i, (F.6b) 

where 0<7r< 1/2, Z^is the degree of variable i and the auxiliary variable is computed 
before updating the field // 

^GO^XV (F.7) 

BP-REINCORCEMENT(T max , n, 7r) 

1 Initialize /x*. and randomly; 
2 

3 Compute the current configuration = argmax s ./i* ; 

4 repeat Make n sweeps of the BP iterations 0F.4allF\4bjl ; 

5 Update all the local fields according to flF.6aHF.6bl) ; 

6 Update = argmax^./x* ; 

7 T<-T+l; 

8 until {r} is a solution or T > T max ; 

How should the strength of the forcing 7r be chosen? Empirically we observed three 
different regimes: 

a) ^BP-iikc < 7r < 0.5: When the forcing is weak the BP-REINFORCEMENT converges 
very fast to a BP-like fixed point, the values of the local fields do not point towards 
any solution. On contrary many constraints are violated by the final configuration 

{n}- 

b) 7r conv < 7T < 7T B p-iikc: The BP-Reinforcement converges to a solution {r,}. 

c) < 7r < 7r conv : When the forcing is too strong the BP-REINFORCEMENT does not 
converge. And many constraints are violated by the configuration {rj} which is 
reached after T max steps. 

When the constraint density in the CSP is large the regime b) disappears and 7r conv = 
TTBP-iike- For an obvious reason our goal is to find 7r conv < n < 7TBP-iikc- The point 7TBp-iiko 
is very easy to find, because for larger 7r the convergence of BP- Reinforcement to a 
BP-like fixed point happens in just several sweeps. Thus in all the runs we chose ir to be 
just bellow 7T B p-iike, that is to hit the possible gap between 7r B p-.iike and vr conv . The value 
of 7r chosen in this way does not seem to depend on the size of the system, it, however, 
depends slightly on the constraint density. 

Experimentally it seems that the optimal number of BP sweeps on line H] of BP- 
Reinforcement is very small, typically n = 2, in agreement with [CFMZ05] . We 
observed with a surprise that when n is much larger not only the total running time is 
larger but the overall performance of the algorithm is worse. 

In the regime where the BP-reinforcement algorithm performs well the median 
running time T seems to be independent of the size, leading to an overall linear time 
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complexity. The total CPU time is comparable to the time achieved by the stochastic 
local search ASAT. 

There is an imperfection of our implementation of the BP-reinforcement, because 
in small fraction of cases, for all connectivities, the algorithm is blocked in a configuration 
with only 1-3 violated constraints. If this happens we reinforce stronger the problematic 
variables which sometimes shifts the problem to a different part of the graph, where it 
might be resolved. Also a restart leads to a solution. 

We tested the BP- Reinforcement algorithm mainly in the occupation CSPs, the 
results are shown in sec. 14.31 Survey propagation reinforcement can be implemented in a 
similar way, as was done originally in [CFMZ05J. 
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REPRINTS OF PUBLICATIONS 



[ZDEB-1] 



"The number of matchings in random 

graphs" 

L. Zdeborova, M. Mezard, J. Stat. Mech, P05003 (2006). 
arXiv:cond-mat / 0603350v2 



This article develops a way how to count matchings in random graphs. We used this 
as an example to introduce the replica symmetric method in sec. 11.5.41 The main result 
of this work is that the belief propagation estimates asymptotically correctly the entropy 
of matchings, this was partially proven on a rigorous level in [BN06J. 
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REPRINT OF PUBLICATION [ZDEB-1] 



[ZDEB-2] 



"A Hike in the Phases of 
the l-in-3 Satisfiability" 

E. Maneva, T. Meltzer, J. Raymond, A. Sportiello, L. Zdeborova, In 
proceedings of the Les Houches Summer School, Session LXXXV 2006 on 

Complex Systems. 

arXiv:cond-mat/0702421vl 



This work on the 1-in-i^ SAT problem started as a student project on the sum- 
mer school in Les Houches 2006: Complex Systems, organized by M. Mezard and J. -P. 
Bouchaud. This short note contains a non-technical overview of our findings, and ap- 
peared in the collection of lecture notes from the school. 
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REPRINT OF PUBLICATION [ZDEB-2] 



[ZDEB-3] 



"The Phase Diagram of l-in-3 Satisfiability" 



J. Raymond, A. Sportiello, L. Zdeborova, Phys. Rev. E 76, 011101 

(2007) 



arXiv:cond-mat /0702610v2 



In this article we present in detail the energetic 1RSB solution of the l-in-3 SAT 
problem. We also analyze the performance of the unit clause propagation algorithms. 
We show how the phase diagram changes from an on average easy to K-SAT like when 
the probability of negating a variable is varied. An interesting point is the existence of a 
region where the replica symmetric solution is unstable, yet the unit clause propagation 
provably finds solutions in a randomized polynomial time. This work is a continuation 
of [ZDEB-2]. We used the 1-in-K SAT to present the energetic 1RSB solution in sec. 11.71 
Note that 1-in-K SAT on factor graphs without leaves is one of the locked problems, this 
article however studies the Poissonian graph ensemble. 
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REPRINT OF PUBLICATION [ZDEB-3] 



[ZDEB-4] 



"Gibbs States and the Set of 
Solutions of Random Constraint 
Satisfaction Problems" 

F. Krzakala, A. Montanari, F. Ricci-Tersenghi, G. Semerjian, L. 
Zdeborova, Proc. Natl. Acad. Set. 104, 10318 (2007). 

arXiv:cond-mat / 0612365v2 



In this article the clustering transition was defined via the extremality of the uniform 
measure over solutions, or equivalently via the onset of a nontrivial solution of the 1RSB 
equations at m = 1. The derivation of the 1RSB equations on trees is sketched. This is 
a basis of our chapter [21 The condensation transition in constraint satisfaction problems, 
different from the clustering one, was discovered here. This is a basis of our chapter [31 
The use of the belief propagation as a solver in the clustered but non-condensed phase 
was suggested here and studied. The results of this short article are developed in greater 
detail in [ZDEB-5] for the graph coloring, and in [MKTS 08] for the K-SAT probl em. This 
article is addressed mainly to a mathematical and computer science audience. 
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[ZDEB-5] 



Phase transition in the Coloring of 
Random Graphs" 



L. Zdeborova, F. Krzakala, Phys. Rev. E 76, 031131 (2007). 



arXiv:0704.1269v2 



This is a detailed article about the phase diagram of the random coloring problem, 
summarized in chapter [5j We give an overview of the entropic 1RSB solution of the 
problem. We derive many results about the clustering and condensation transitions. 
The cavity method study of the frozen variables, as presented in 14.21 is developed here. 
The conjecture about freezing of variables being relevant for the computational hardness, 
which we discuss in 14.41 is made here. 
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REPRINT OF PUBLICATION [ZDEB-5] 



[ZDEB-6] 



"Potts Glass on Random Graphs" 

F. Krzakala, L. Zdeborova, Eur. Phys. Lett. 81 (2008) 57005. 

arXiv:0710.3336v2 



In this letter we present the finite temperature phase diagram of the coloring problem, 
or in other words the antiferromagnetic Potts model on random graphs. We showed the 
phase diagram in sec. 15.41 We also analyze the stability of the 1RSB solution and in 
particular show that the colorable phase is 1RSB stable (at least for q > 4). This is 
reviewed in more detail in appendix ID1 
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[ZDEB-7] 



"Phase Transitions and Computational 
Difficulty in Random Constraint 
Satisfaction Problems" 

F. Krzakala, L. Zdeborova, J. Phys.: Conf. Ser. 95 (2008) 012012. 



arXiv:0711.0110vl 



In this article we present in a accessible and non-technical way the main new results 
for the phase diagram of the random coloring. This might be a good reading for unini- 
tialized audience. We also summarize the present ideas about the origin of the average 
computational hardness. This article was presented in the Proceedings of the Interna- 
tional Workshop on Statistical-Mechanical Informatics, Kyoto 2007. Chapter [5] is largely 
inspired by this colloquial presentation. 
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[ZDEB-8] 



"Random subcubes as a toy model for 
constraint satisfaction problems" 

T. Mora, L. Zdeborova, J. Stat. Phys. 131 n.6 (2008) 1121-1138. 

arXiv:0710.3804v2 



In this article we introduced the random subcubes model. It plays the same role for 
constraint satisfaction problems as the random energy model played for spin glasses. It 
is an exactly solvable toy model which reproduces the series of phase transitions studied 
in CSPs. The condensation transition comes out very naturally in this simple model, see 
sec. 13 . 1 L The space of solutions in the random subcubes model compares even quantita- 
tively to the space of solutions in the g-coloring and if-SAT in the limit of large q and 
K near to the satisfiability threshold, as explained in sec. 15.3.11 We also introduced an 
energy landscape and showed that the glassy dynamics in this model can be understood 
purely from the static solution. 
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[ZDEB-9] 



"Locked constraint satisfaction problems 

L. Zdeborova, M. Mezard, to be accepted in Phys. Rev. Lett. 

arXiv:0803.2955vl 



In this letter we introduce the locked constraint satisfaction problems, presented in 
sec. 14.31 The space of solutions of these problems have an extremely easy statistical 
description, as illustrated e.g. by the second moment computation of the entropy in 
app. [HI On the other hand these problems are algorithmically very challenging, none of 
the algorithms we tried is able to find solutions in the clustered phase. Some classical 
algorithms do not work at all in these problems, for example the belief propagation 
decimation analyzed in app. [F] A more detailed version of this article is in preparation. 
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[ZDEB-10] 



"Exhaustive enumeration unveils clustering 
and freezing in random 3-SAT" 

J. Ardelius, L. Zdeborova, submitted to Phys. Rev. 



arXiv:0804.0362v2 



In this letter we study via an exhaustive enumeration the phase space in the random 3- 
SAT. The main question we addressed here is the relevance of the asymptotic predictions 
to instances of moderate size. We show that the complexity of clusters compares strikingly 
well to the analytical prediction. We also locate for a first time the freezing transition and 
show that it practically coincides with the performance limit of the survey propagation 
based algorithms. Results of this work appear on several places of the thesis, mainly 
figs. EH [£21 and El 
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